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(U)  In  an  earlier  pilot  study  of  the  narrative  sections  of  Navy  performance  evalu¬ 
ations  for  senior  enlisted  personnel  in  pay  grade  E-7,  it  was  determined  by  content 
analytic  techniques  that  it  is  possible  to  differentiate  between  the  performance  of 
typical  and  superlative  chief  petty  officers  based  on  the  narrative  content  of  Evalua¬ 
tion  Reports.  A  second  study  attempted  to  cross  validate  the  pilot  study  results  on 
new  Evaluation  Reports  for  senior  enlisted  men  in  the  same  two  occupational  ratings 
(ATfs  and  BTfs)  represented  in  the  pilot  study  sample  and  to  extend  the  content  analy¬ 
sis  to  two  different  occupational  ratings  (CSfs  and  RMfs)  in  order  to  test  the  general- 
izability  of  the  content  analytic  techniques  developed  earlier.  As  a  further  refine¬ 
ment,  the  cross  validation  and  generalization  samples  were  analyzed  without  any  know¬ 
ledge  of  the  rateeTs  relative  position  in  the  upper  half  of  the  marking  scale  on  Per¬ 
formance  of  Duty  (the  criterion  variable).  A  reliability  study  was  also  conducted 
whose  objectives  were  twofold:  (1)  to  determine  the  level  of  agreement  among  four  in¬ 
dividuals  all  of  whom  independently  would  perform  a  content  analysis  of  the  same  corpus 
of  48  Evaluation  Reports,  and  (2)  to  investigate  if  nonresearchers  could  be  trained 
successfully  to  apply  the  complex  content  analysis  methodology  developed  in  the  pilot 
study.  This  report  presents  detailed  evidence  that  the  pilot  study  findings  essen¬ 
tially  were  extendible  to  the  blind  cross  validation  sample  consisting  of  the  same  two 
ratings  and  also  were  generalizable  to  two  different  occupational  ratings.  Despite  the 
initial  expectation  that  it  would  be  extremely  difficult  to  train  nonresearch-oriented 
individuals  to  consistently  index  the  narrative  sections  of  Evaluation  Reports,  the 
surprising  result  in  the  reliability  study  was  that  in  only  six  training  sessions  a 
quite  respectable  level  of  agreement  was  achieved. 
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SUMMARY  OF  FINDINGS 

In  an  earlier  pilot  study  of  the  narrative  sections  of  Navy  performance 
evaluations  for  senior  enlisted  personnel  in  pay  grade  E-7,  it  was  determined 
by  content  analytic  techniques  that  it  is  possible  to  differentiate  between 
the  performance  of  typical  and  superlative  chief  petty  officers  based  on  the 
substantive  content  of  Evaluation  Reports.  The  results  of  this  pilot  study 
strongly  suggested  that  there  are  stable  differences  among  the  performance 
characteristics  of  chief  petty  officers  in  the  various  portions  of  the  upper 
half  of  the  marking  scale  on  Performance  of  Duty  that  are  reflected  in  nar¬ 
rative  statements  written  by  evaluators. 

The  findings  from  the  pilot  study  were  considered  to  be  provocative 
enough  to  warrant  further  investigation.  Therefore,  a  second  study  was  em¬ 
barked  upon  to  attempt  to  cross  validate  the  pilot  study  results  on  new  Evalua¬ 
tion  Reports  for  senior  enlisted  men  in  the  same  two  occupational  ratings  (AT’s 
and  BT’s)  that  were  represented  in  the  pilot  study  sample  and  to  extend  the 
content  analysis  to  Evaluation  Reports  for  senior  enlisted  men  in  two  differ¬ 
ent  occupational  ratings  (CS’s  and  RMfs)  than  those  investigated  in  the  pilot 
study  in  order  to  test  the  generalizability  of  the  content  analytic  techniques 
developed  earlier.  As  a  further  refinement,  the  cross  validation  and  gener¬ 
alization  samples  of  Evaluation  Reports  were  to  be  analyzed  without  any  knowl¬ 
edge  of  the  ratee’s  relative  position  in  the  upper  half  of  the  marking  scale 
on  Performance  of  Duty  (the  criterion  variable).  In  the,  pilot  study  the  cri¬ 
terion  data  were  made  available  early  in  the  study,  thus  introducing  the  pos¬ 
sibility  that  this  knowledge  subconsciously  might  have  influenced  the  content 
analysis  that  was  performed.  This  factor  was  controlled  for  in  the  second 
study  by  withholding  the  criterion  information  until  the  content  analysis  of 
the  narrative  text  had  been  completed. 

Also  of  concern  in  the  pilot  study  were  the  issues  of  reliability  and 
trainability,  although  the  scope  of  the  small  initial  research  effort  did  not 
permit  these  aspects  to  be  studied  in  any  substantial  way.  Therefore,  in  de¬ 
signing  the  second  investigation  these  issues  were  dealt  with  by  including  a 
reliability  study  whose  objectives  were  twofold:  (1)  to  determine  the  level 
of  agreement  among  four  individuals  all  of  whom  independently  would  perform  a 
content  analysis  of  the  same  corpus  of  48  Evaluation  Reports,  and  (2)  to  in¬ 
vestigate  if  nonresearchers  could  be  trained  successfully  to  apply  the  complex 
content  analysis  methodology  developed  in  the  pilot  study. 

In  the  earlier  pilot  investigation,  the  Navy  Personnel  Research  and 
Development  Center  (NPRDC),  San  Diego,  selected  a  sample  of  225  Evaluation  Re¬ 
ports  for  senior  enlisted  personnel  in  pay  grade  E-7  including  145  Aviation 
Electronics  Technicians  (AT’s)  and  80  Boilermen  (BT’s).  All  225  Evaluation 
Reports  were  drawn  from  the  top  half  of  the  marking  scale  on  19A-PERF0RMANCE 
OF  DUTY  of  Evaluation  Report  Form  NAVPERS  1616/8.  This  form  subsequently  has 
been  replaced  by  another  form  that  can  be  scanned  by  an  optical  character 
reader;  however,  the  content  of  the  two  forms  is  essentially  the  same.  The 
pilot  study  sample  of  225  Evaluation  Reports  was  divided  equally  into  three 
criterion  groups - Upper,  Middle,  and  Lower - corresponding  to  three  continu¬ 

ous  segments  of  the  upper  half  of  the  marking  scale  on  19A-PERF0RMANCE  OF 
DUTY. 
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In  the  study  being  reported  here,  NPRDC  also  selected  the  sample  of  Evalua¬ 
tion  Reports  to  be  analyzed.  The  cross  validation  sample  consisted  of  222 
Evaluation  Reports  from  the  same  two  ratings  that  were  used  in  the  pilot  study 
(i.e.,  AT’s  and  BT’s).  In  addition,  a  generalization  sample  consisting  of  222 
Evaluation  Reports  was  also  selected  by  NPRDC  from  two  different  ratings  in 
order  to  ascertain  the  generalizability  of  the  content  analytic  methodology 
developed  in  the  pilot  study.  The  two  ratings  from  which  the  generalization 
sample  was  drawn  were  Commissarymen  (OS’s)  and  Radiomen  (RM’s).  The  Nfs  in 
the  various  occupational  ratings  represented  in  the  cross  validation  and  gen¬ 
eralization  samples  were  138  AT?s,  84  BTfs,  60  CS’s,  and  162  RM’s.  The  cross 
validation  sample  and  the  generalization  sample  were  each  divided  equally  into 

the  same  three  criterion  groups  as  the  pilot  study  sample - Upper,  Middle,  and 

Lower.  Actual  criterion  group  membership  for  the  cross  validation  sample  and 
the  generalization  sample  was  known  only  to  NPRDC  until  the  content  analysis 
of  the  narrative  text  had  been  completed.  Consequently,  the  content  analysis 
of  these  two  samples  was  conducted  in  the  blind  without  benefit  of  knowing  to 
which  criterion  group  each  Evaluation  Report  belonged. 

An  indexing  vocabulary  consisting  of  29  descriptive  labels  was  devised  to 
encompass  the  substantive  content  of  the  narrative  sections  of  Evaluation  Re¬ 
ports.  These  29  index  terms  fell  into  three  major  areas - MANAGEMENT  FUNCTIONS, 

SKILLS  AND  ABILITIES,  and  PRODUCTIVITY  AND  ACHIEVEMENT.  Under  each  of  these 
headings  there  were  more  detailed  terms  such  as  PLANNING,  TECHNICAL  SKILLS, 
and  AWARDS  AND  PUNISHMENT,  providing  the  indexer  with  a  3-level  hierarchy  of 
descriptive  labels  from  which  to  choose.  Each  sentence  of  narrative  text  in 
the  pilot  study  sample  and  the  cross  validation  and  generalization  samples  was 
read  carefully  and,  where  appropriate,  divided  into  segments  corresponding  to 
the  assignment  of  specific  index  terms.  However,  it  is  not  enough  to  simply 
label  a  narrative  statement  with  the  most  appropriate  index  term  since  the 
statement  may  have  been  a  highly  positive,  quite  positive,  neutral,  quite  nega¬ 
tive,  or  highly  negative  one.  Therefore,  a  weighting  scale  containing  five 
degrees  of  favorableness/unfavorableness  was  devised  based  on  the  range  of  ad¬ 
jectives  and  adverbs  that  occur  in  narrative  text  of  this  kind.  The  indexing 
procedure  that  was  used  in  this  study  was  the  following:  The  narrative  text 
of  each  Evaluation  Report  was  read,  segmented  into  distinct  statements,  and 
each  statement  was  assigned  one  or  more  index  terms  from  the  set  of  29  pos¬ 
sible  choices.  Each  term  selected  was  also  assigned  a  numerical  weight  from 
1  to  5  depending  upon  the  nature  of  the  adjectives  or  adverbs  used  as  modi¬ 
fiers  in  the  statement.  When  the  entire  narrative  text  of  the  Evaluation  Re¬ 
port  had  been  indexed,  the  indexing  decisions  that  had  been  made  were  record¬ 
ed  on  a  special  indexing  form. 

A  set  of  67  quantitative  variables  was  derived  from  the  indexing  form 
used  in  the  content  analysis.  The  first  29  variables  reflect  the  simple  fre¬ 
quency  with  which  each  index  term  was  used  to  index  a  particular  section  of 
narrative  text.  Variable  30  is  the  sum  of  these  29  frequencies.  Variables 
30  through  59  represent  the  weighted  frequency  of  each  index  term  used  to 
index  a  particular  section  of  narrative  text.  Variable  60  is  similar  to  Varia¬ 
ble  30  in  that  it  is  the  sum  of  the  29  weighted  frequencies.  Variables  61 
through  65  represent  the  frequency  counts  over  the  entire  indexing  form  for 
all  5  weights,  4  weights,  3  weights,  2  weights,  and  1  weights.  Variable  66  is 
the  total  number  of  words  in  the  section  of  narrative  text  that  was  indexed. 
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Variable  67  is  the  total  number  of  index  terms  of  the  29  available  that  were 
used  to  index  the  section  of  narrative  text.  Profiles  or  vectors  of  these 
67  values  then  were  prepared  for  all  of  the  Evaluation  Reports  contained  in 
each  sample.  Separate  profiles  were  compiled  for  the  evaluation  section  (19R) 
and  the  justification  section  (19S)  of  each  Evaluation  Report. 

Descriptive  statistics  were  computed  for  each  of  the  three  research  sam¬ 
ples  on  the  67  quantitative  variables.  As  expected  these  statistics  showed 
in  general  that  the  higher  the  criterion  group,  the  longer  the  narrative  text. 
Also,  as  the  evaluator  uses  more  words  to  describe  the  ratee,  he  is  more  like¬ 
ly  to  comment  on  a  wider  variety  of  specific  areas  of  the  ratee rs  performance. 
Correlations  among  all  67  variables  for  the  evaluation  section  and  for  the 
justification  section  also  were  computed  for  the  cross  validation  and  generali¬ 
zation  samples  as  well  as  the  matrix  correlating  the  evaluation  section  with 
the  justification  section  on  all  67  variables.  There  were  very  few  high  cor¬ 
relations  except  for  the  correlations  between  Variables  1  through  30  and 
Variables  31  through  60,  these  two  sets  of  variables  being  the  same  except  for 
the  method  of  weighting  that  was  used.  Variable  67  (Total  Number  of  Index 
Terms  Used)  is  a  focal  variable,  correlating  highly  with  Variable  30  (Sum  of 
Variables  1  through  29),  Variable  60  (Sum  of  Variables  31  through  59),  Varia¬ 
ble  61  (Total  Number  of  5  Weights),  Variable  62  (Total  Number  of  4  Weights), 
Variable  63  (Total  Number  of  3  Weights),  and  Variable  66  (Total  Number  of 
Words  in  the  Narrative  Text).  Correlations  among  the  variables  having  to  do 
with  the  29  index  terms  per  se  were  notably  low,  indicating  that  these  29  di¬ 
mensions  are  relatively  orthogonal  and  represent  independent  aspects  of  mana¬ 
gerial  performance.  In  the  correlation  matrix  for  the  evaluation  section  ver¬ 
sus  the  justification  section,  no  high  correlations  were  evident  in  either  the 
cross  validation  sample  or  the  generalization  sample,  demonstrating  that  these 
two  narrative  sections  of  Evaluation  Report  Form  NAVPERS  1616/8  are  quite  in¬ 
dependent  and  should  be  treated  separately. 

Both  the  Mann-Whitney  U  test  and  the  t  test  of  mean  difference  were  com¬ 
puted  on  each  of  the  67  variables  for  the  pilot  study  sample,  the  cross  valida¬ 
tion  sample,  and  the  generalization  sample.  These  computations  were  made  for 
each  pair  of  criterion  groups  in  both  the  evaluation  and  the  justification  sec¬ 
tions.  The  most  difficult  discrimination  to  be  made  is  that  between  the  Mid¬ 
dle  and  Upper  criterion  groups.  In  the  cross  validation  sample  the  character¬ 
istics  that  differentiated  outstanding  CPO's  from  their  slightly  less  quali¬ 
fied  colleagues  on  the  evaluation  section  were  cooperation,  grooming  and  at¬ 
tire,  resourcefulness,  and  productivity  and  achievement.  Cooperation  was  also 
implicated  as  a  discriminating  variable  between  the  Middle  and  Upper  criterion 
groups  on  the  evaluation  section  in  the  pilot  study  sample.  On  the  justifica¬ 
tion  section,  eight  variables  showed  a  statistically  significant  difference 
between  the  Middle  and  Upper  criterion  groups  in  both  the  pilot  study  sample 
and  the  cross  validation  sample.  When  an  evaluator  is  required  to  justify 
his  marks  in  evaluating  a  ratee,  he  apparently  calls  out  certain  areas  of 
performance  that  distinguish  the  ratee  in  the  Upper  criterion  group  from  his 
slightly  less  qualified  colleague  in  the  Middle  criterion  group.  Skills  and 
abilities  as  well  as  productivity  and  achievement  were  the  differentiating 
areas  of  performance.  The  ratee  in  the  Upper  criterion  group  also  had  be¬ 
stowed  upon  him  more  superlative  adjectives  and  adverbs.  Total  Number  of  Words 
in  Text  and  Total  Number  of  Index  Terms  Used  were  also  discriminating  variables. 
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In  the  comparison  between  the  Middle  and  Upper  criterion  groups  on  the 
evaluation  section  of  the  generalization  sample.  Total  Number  of  5  Weights 
(Excellent),  Total  Number  of  2  Weights  (Poor),  and  potential  were  the  discrimi¬ 
nating  variables.  None  of  these  variables  overlapped  with  those  that  were 
statistically  significant  for  the  Middle  versus  Upper  criterion  group  compari¬ 
son  on  the  evaluation  section  of  the  cross  validation  sample.  Seventeen  of 
the  67  quantitative  variables  showed  a  statistically  significant  difference  be¬ 
tween  the  Upper  and  Middle  criterion  groups  on  the  justification  section  for 
both  the  cross  validation  sample  and  the  generalization  sample,  the  most  sig¬ 
nificant  of  which  were  Total  Number  of  5  Weights  (Excellent),  Total  Number  of 
4  Weights  (Good),  Total  Number  of  2  Weights  (Poor),  Total  Number  of  Index 
Terms  Used,  Sum  of  Variables  1  through  29,  Sum  of  Variables  31  through  59, 
leadership  and  directing,  communication,  cooperation,  technical  skills,  drive, 
and  potential.  This  finding  suggests  that  there  is  partial  overlap  between 
these  two  samples.  The  nonoverlapping  areas  may  be  attributed  to  a  difference 
in  the  nature  of  the  occupational  ratings  represented  in  the  cross  validation 
and  the  generalization  samples  and/or  to  unreliability  in  the  indexing  pro¬ 
cedure,  although  the  results  of  the  reliability  study  suggest  that  differences 
among  the  four  occupational  ratings  provide  a  more  reasonable  explanation  for 
these  results  than  unreliability. 

In  every  pairwise  criterion  group  comparison  made  for  the  three  research 
samples  using  the  Mann-Whitney  U  test  and  the  t  test  of  mean  difference,  one 
or  more  of  the  five  variables  involving  total  number  of  weights  proved  to  be 
significantly  differentiating.  This  finding  supports  the  results  of  the  cor¬ 
relational  analysis  in  which  the  system  used  for  weighting  Variables  31  through 
59  was  highlighted  as  an  important  aspect  of  the  content  analysis  methodology. 

It  is  of  considerable  interest  to  learn  how  well  the  set  of  67  quantita¬ 
tive  variables,  used  in  optimal  combination,  can  classify  each  of  the  research 
samples  into  correct  criterion  group.  A  stepwise  discriminant  analysis  pro¬ 
gram  was  used  to  perform  this  analysis.  A  special  feature  of  this  program 
allows  new  cases  to  be  classified  by  the  discriminant  functions  generated  on 
the  original  sample.  This  feature  was  used  to  conduct  two  cross  validation 
studies  of  the  ATfs  and  BT’s  combined  and  also  of  the  AT’s  and  BT’s  considered 
separately.  The  results  of  the  two  cross  validation  studies  were  very  similar 
for  the  total  cross  validation  sample  and  the  total  pilot  study  sample,  for 
the  cross  validation  AT’s  and  the  pilot  study  AT’s,  and  for  the  cross  valida¬ 
tion  BT’s  and  the  pilot  study  BT’s.  These  findings  support  the  expectation 
held  at  the  outset  of  this  investigation  that  it  would  be  possible  to  index 
the  cross  validation  sample  in  the  blind,  without  knowledge  of  criterion  group 
membership,  and  achieve  as  good  classification  accuracy  as  was  achieved  with 
the  pilot  study  sample  where  criterion  group  membership  was  known  to  the  in¬ 
dexer.  Further,  it  can  be  concluded  that  better  classification  into  the 
three  criterion  groups  using  an  optimum  combination  of  the  67  quantitative 
variables  is  achieved  when  the  two  occupational  ratings  represented  in  the  pi¬ 
lot  study  sample  and  the  cross  validation  sample  are  treated  separately. 

These  findings  suggest  that  classification  procedures  based  on  the  content 
analysis  methodology  developed  in  this  research  should  be  tailored  to  specific 
occupations.  In  all  of  the  stepwise  discriminant  analyses  performed,  better 
classification  was  achieved  in  the  analysis  of  the  justification  section  com¬ 
pared  to  the  evaluation  section.  Classification  of  each  sample  by  its  own 
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discriminant  functions  achieved  perfect  classification  for  the  BT’s  on  the 
justification  section  in  both  the  pilot  study  sample  and  the  cross  validation 
sample.  In  the  pilot  study  sample,  95  percent  of  the  ATfs  were  classified 
correctly  on  the  justification  section.  In  the  cross  validation  sample,  93 
percent  of  the  AT?s  were  classified  correctly  on  the  justification  section. 

The  superior  classification  accuracy  achieved  for  the  BT's  compared  to  the 
AT?s  indicates  that  the  Aviation  Electronics  Technician  rating  may  represent 
a  more  varied  amalgamation  of  technical  activities  than  the  Boilerman  rating. 
Further,  it  appears  that  the  best  classification  accuracy  than  can  be  achieved 
on  a  second  sample  using  the  discriminant  functions  generated  on  the  first 
sample,  with  the  content  analysis  methodology  developed  thus  far,  is  65  to  70 
percent . 

In  the  stepwise  discriminant  analyses  of  the  generalization  sample,  all 
60  generalization  CS's  were  correctly  classified  on  the  justification  section. 
Of  the  162  generalization  RMfs,  89  percent  were  correctly  classified  on  the 
justification  section.  This  suggests  that  the  occupational  rating,  Radioman, 
may  be  more  heterogeneous  and  require  a  greater  variety  of  skills  than  the 
Commissaryman  rating  just  as  the  Aviation  Electronics  Technician  rating  may 
represent  a  more  varied  amalgamation  of  technical  activities  than  the  Boiler- 
man  rating.  It  is  evident  from  these  results  that  the  content  analysis  method¬ 
ology  developed  initially  on  the  pilot  study  sample  consisting  of  AT's  and 
BT’s  was  generalizable  to  a  new  sample  consisting  of  two  different  occupation¬ 
al  ratings,  viz.,  CSfs  and  RMfs. 

In  the  double  cross  validation  of  the  pilot  study  sample  and  the  cross 
validation  sample,  the  best  classification  accuracy  for  the  sample  being 
cross  validated  was  achieved  early  in  the  stepwise  discriminant  analysis  pro¬ 
cedure,  typically  by  the  fifth  step.  The  key  discriminating  variables  for 
the  evaluation  section  were  Total  Number  of  5  Weights  (Excellent)  and  Total 
Number  of  2  Weights  (Poor).  In  the  justification  section  without  exception 
the  key  discriminating  variable  was  Total  Number  of  Index  Terms  Used.  These 
same  three  variables  were  those  selected  first  in  the  stepwise  discriminant 
analysis  of  the  generalization  sample.  It  appears  that  the  modifying  adjec¬ 
tives  used  by  an  evaluator  to  rate  a  ratee  and  the  range  of  skills  and  abili¬ 
ties  that  a  chief  petty  officer  possesses  may  be  key  factors  in  the  ratee1 s 
superior  performance.  The  results  also  suggest  that  a  smaller  number  of  di¬ 
mensions  than  the  full  complement  of  67  quantitative  variables  derived  from 
the  indexing  procedure  can  be  used  to  identify  superlative  CPO’s  whose  supe¬ 
rior  performance  recommends  them  as  candidates  for  promotion  to  a  higher  level 
of  responsibility. 

In  addition  to  the  cross  validation  and  generalization  study,  a  compre¬ 
hensive  reliability  study  was  conducted  whose  objectives  were  twofold:  (1) 
to  determine  the  level  of  agreement  among  several  individuals  all  of  whom 
independently  would  perform  a  content  analysis  of  the  same  corpus  of  Evalua¬ 
tion  Reports,  and  (2)  to  investigate  if  nonresearchers  could  be  trained  suc¬ 
cessfully  to  apply  the  complex  content  analysis  methodology  developed  in  the 
pilot  study. 

A  set  of  48  Evaluation  Reports  was  selected  by  the  Navy  Personnel  Research 
and  Development  Center,  representing  a  cross  section  of  the  kinds  of  reports 
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included  in  the  overall  experimental  design  for  the  cross  validation  and  gener¬ 
alization  samples.  In  each  of  these  48  Evaluation  Reports  the  evaluation  sec¬ 
tion  was  separated  from  the  justification  section  so  that  the  narrative  comments 
for  each  section  were  not  considered  together.  This  resulted  in  a  group  of  96 
randomized  pieces  of  narrative  text  to  be  indexed  in  the  reliability  study. 

Four  individuals  participated  in  the  reliability  study:  (1)  the  experi¬ 
enced  indexer  who  also  indexed  the  pilot  study  sample,  the  cross  validation 
sample,  and  the  generalization  sample;  (2)  the  principal  investigator;  (3)  an 
inexperienced  indexer  (inexperienced  indexer  A)  with  two  years  of  college  in 
the  liberal  arts;  and  (4)  another  inexperienced  indexer  (inexperienced  indexer 
B)  with  executive  secretary  experience.  To  this  end  a  training  manual  was 
prepared  by  the  experienced  indexer  and  the  principal  investigator  to  assist 
the  two  neophyte  indexers  in  understanding  their  assignment.  Six  intensive 
training  sessions  were  conducted  by  the  experienced  indexer  in  order  to  try 
to  bring  all  four  indexers  up  to  a  common  level  of  expertise  before  beginning 
the  actual  study.  Obviously,  this  objective  could  only  be  met  partially  in 
view  of  the  varying  educational  backgrounds  of  the  four  reliability  indexers 
and  their  different  levels  of  previous  exposure  to  the  indexing  dictionary, 

In  all  of  the  agreement  statistics  that  were  computed,  assignment  of  the 
index  terms  was  considered  to  be  a  separate  intellectual  task  from  assigning 
the  corresponding  weights  based  on  the  modifying  adjectives  and  adverbs.  The 
kappa  statistic  was  the  measure  of  agreement  used  in  analyzing  the  index  terms 
assigned  by  the  four  reliability  indexers.  The  best  agreement  in  selecting 
index  terms  was  obtained  between  the  experienced  indexer  and  inexperienced  in¬ 
dexer  A,  a  kappa  of  .88  where  the  maximum  kappa  possible  in  this  instance  was 
.97.  Of  the  six  possible  pairwise  comparisons  between  the  four  reliability 
indexers,  the  value  of  kappa  ranged  from  .71  to  .88,  with  .71  probably  rep¬ 
resenting  the  lower  limit  of  reliability  achievable  in  a  study  of  this  kind. 

The  kappa  analysis  revealed  that  the  major  area  of  confusion  in  indexing  the 
reliability  data  base  resided  in  whether  or  not  to  index  supposedly  factual 
statements  describing  the  job  duties  and  the  qualifications  needed  for  the 
position  that  the  ratee  occupied  rather  than  the  ratee 's  actual  performance  in 
this  position.  All  three  of  the  less  experienced  indexers  tended  to  index 
these  statements  as  describing  the  ratee fs  performance  whereas  the  experienced 
indexer  whom  the  other  three  indexers  were  trying  to  emulate  treated  these 
statements  as  factual  descriptions  of  the  job  duties  and  the  qualifications 
needed  for  the  position.  Additional  training  aimed  at  clarifying  this  area 
of  confusion  most  likely  would  markedly  reduce  this  type  of  disagreement  and 
raise  the  magnitude  of  kappa. 

Analysis  of  the  level  of  agreement  among  the  four  reliability  indexers  in 
assigning  numerical  weights  to  each  index  term  selected,  based  on  the  modify¬ 
ing  adjectives  and  adverbs,  was  performed  differently  than  the  analysis  of  the 
level  of  agreement  in  selecting  the  index  terms  themselves,  because  the  numeri¬ 
cal  weights  assigned  to  the  index  terms  constituted  an  ordinal  scale  whereas 
the  index  terms  themselves  formed  a  nominal  scale.  In  the  six  pairwise  com¬ 
parisons  between  the  four  reliability  indexers,  six  product  moment  correlation 
coefficients  were  computed  as  well  as  another  agreement  statistic,  weighted 
kappa,  in  order  to  determine  if  weighted  kappa  agreed  with  the  results  of  the 
correlational  analysis.  In  the  correlational  analysis,  once  again  the  best 
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agreement  in  assigning  numerical  weights  to  each  index  term  selected  was  obtained 
between  the  experienced  indexer  and  inexperienced  indexer  A,  a  correlation  co¬ 
efficient  of  .80.  These  findings  corroborate  each  other  in  suggesting  that  an 
individual  without  a  research  background  in  only  six  training  sessions  can  be 
taught  not  only  how  to  select  the  most  appropriate  index  terms  but  also  how  to 
consistently  assign  weights  to  these  terms  based  on  the  modifying  adjectives 
and  adverbs.  The  other  five  correlation  coefficients  were  lower,  but  none 
less  than  .64. 

As  was  expected,  the  weighted  kappa  values  were  similar  in  magnitude  to 
their  correlation  coefficient  counterparts.  Again,  the  best  agreement  as 
measured  by  weighted  kappa  was  obtained  between  the  experienced  indexer  and  in¬ 
experienced  indexer  A,  a  weighted  kappa  of  .78.  If  the  area  of  confusion  in¬ 
volving  overindexing  on  the  part  of  inexperienced  indexer  A  was  ignored  in  the 
analysis,  the  value  of  weighted  kappa  increased  to  .82.  The  gain  in  the  value 
of  weighted  kappa  is  not  very  large  for  the  comparison  between  the  experienced 
indexer  and  inexperienced  indexer  A  when  weighted  kappa  was  recomputed  in  this 
fashion.  However,  the  gain  was  quite  substantial  in  the  other  comparisons  be¬ 
tween  the  experienced  indexer  and  the  principal  investigator  and  between  the 
experienced  indexer  and  inexperienced  indexer  B.  This  suggests  that  with  ad¬ 
ditional  training  to  clarify  this  area  of  confusion  and  with  more  indexing 
experience,  the  level  of  agreement  among  the  four  reliability  indexers  could 
possibly  be  raised  to  a  value  of  .80  to  .85  as  measured  by  any  of  the  three 
agreement  statistics  employed  in  this  study.  However,  values  in  the  .90’s 
are  the  ultimate  objective. 

In  conclusion,  it  might  be  of  interest  to  point  out  that  the  initial  ex¬ 
pectation  in  beginning  this  reliability  study  was  that  it  would  be  extremely 
difficult  to  train  nonresearch-oriented  individuals  to  consistently  index  the 
narrative  sections  of  Evaluation  Report  forms  using  the  complex  content  analy¬ 
sis  methodology  that  had  been  developed  in  the  pilot  study.  The  surprising 
result  is  that  in  only  six  training  sessions  a  quite  respectable  level  of 
agreement  was  achieved.  Moreover,  one  of  the  inexperienced  indexers  showed 
a  higher  level  of  agreement  with  the  experienced  indexer  than  the  principal 
investigator  did,  and  the  other  inexperienced  indexer  agreed  with  the  experi¬ 
enced  indexer  almost  as  well  as  the  principal  investigator.  The  intuitive 
feeling  that  the  reliability  indexers  had  after  completing  the  reliability 
study  was  that  the  most  difficult  part  of  learning  to  index  consistently  was 
over  and  that  with  additional  practice  and  some  review  training  sessions  they 
could  improve  their  indexing  skill. 


1 


SECTION  1.  INTRODUCTION 

A  goal  of  on-going  research  being  conducted  by  the  Navy  Personnel  Re¬ 
search  and  Development  Center,  San  Diego,  is  to  develop  Navy  enlisted  perform¬ 
ance  evaluation  formats  which  will  be  effective  in  holding  down  the  pile-up  of 
marks  at  the  high  end  of  the  marking  scale  and  in  achieving  a  distribution  of 
marks  which  tapers  off  sufficiently  at  the  high  end  of  the  scale  to  permit 
greater  differentiation  among  ratees,  making  evaluations  more  useful,  especial¬ 
ly  when  small  selection  opportunities  are  involved.1  Thus  far  the  narrative 
sections  of  Evaluation  Reports  have  not  been  exploited  to  any  great  extent  in 
the  design  of  experimental  forms  because  narrative  text  tends  to  resist  easy 
analysis.  However,  in  a  pilot  investigation  of  the  narrative  sections  of  Navy 
performance  evaluations  for  senior  enlisted  personnel  in  pay  grade  E-7  con¬ 
ducted  by  R-K  Research  and  System  Design,  it  was  determined  by  content  analytic 
techniques  that  it  is  possible  to  differentiate  between  the  performance  of 
typical  and  superlative  chief  petty  officers  based  on  the  substantive  content 
of  Evaluation  Reports.2  The  results  of  this  pilot  study  strongly  suggested 
that  there  are  stable  differences  among  the  performance  characteristics  of 
chief  petty  officers  in  the  various  portions  of  the  upper  half  of  the  marking 
scale  on  Performance  of  Duty  that  are  reflected  in  narrative  statements  writ¬ 
ten  by  evaluators.  These  differences  are  both  identifiable  and  quantifiable. 

In  the  pilot  study  the  significant  differences  resided  in  the  superiority  of 
the  uppermost  criterion  group  with  respect  to  managerial  skills  and  abilities, 
particularly  as  demonstrated  in  the  areas  of  organization,  initiative,  coop¬ 
eration,  leadership  and  directing,  professionalism,  productivity  and  achieve¬ 
ment,  more  awards  and  fewer  punishments,  more  drive,  more  superlative  attri¬ 
butes,  and  fewer  qualified  statements  reflecting  fair  performance. 

The  findings  from  the  pilot  study  were  considered  to  be  provocative 
enough  to  warrant  further  investigation.  Therefore,  a  second  study  was  em¬ 
barked  upon  to  attempt  to  cross  validate  the  pilot  study  results  on  new  Evalua¬ 
tion  Reports  for  senior  enlisted  men  in  the  same  two  occupational  ratings  that 
were  represented  in  the  pilot  study  sample  and  to  extend  the  content  analysis 
to  Evaluation  Reports  for  senior  enlisted  men  in  two  different  occupational 
ratings  than  those  investigated  in  the  pilot  study  in  order  to  test  the  gener- 
alizability  of  the  content  analytic  techniques  developed  earlier.  As  a  fur¬ 
ther  refinement,  the  cross  validation  and  generalization  samples  of  Evaluation 
Reports  were  to  be  analyzed  without  any  knowledge  of  the  rateefs  relative  posi¬ 
tion  in  the  upper  half  of  the  marking  scale  on  Performance  of  Duty  (the  crite¬ 
rion  variable).  In  the  pilot  study  the  criterion  data  were  made  available 
early  in  the  study,  thus  introducing  the  possibility  that  this  knowledge  sub¬ 
consciously  might  have  influenced  the  content  analysis  that  was  performed  on 
the  narrative  sections  of  the  Evaluation  Reports.  This  factor  was  controlled 
for  in  the  second  study  by  withholding  the  criterion  information  until  the  con¬ 
tent  analysis  of  the  narrative  text  had  been  completed. 

Also  of  concern  in  the  pilot  study  were  the  issues  of  reliability  and 
trainability ,  although  the  scope  of  the  small  initial  research  effort  did  not 
permit  these  aspects  to  be  studied  in  any  substantial  way.  Therefore,  in  de¬ 
signing  the  second  investigation  these  issues  were  dealt  with  by  including  a 
reliability  study  whose  objectives  were  twofold:  (1)  to  determine  the  level 
of  agreement  among  four  individuals  all  of  whom  independently  would  perform  a 
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content  analysis  of  the  same  corpus  of  Evaluation  Reports,  and  (2)  to  investi¬ 
gate  if  nonresearchers  could  be  trained  successfully  to  apply  the  complex  con¬ 
tent  analysis  methodology  developed  in  the  pilot  study. 

The  second  study  attempting  to  cross  validate  and  generalize  the  pilot 
study  results  and  to  elucidate  the  issues  of  reliability  and  trainability  was 
conducted  during  the  contract  year  March  1,  1972  to  February  28,  1973,  This 
technical  report  presents  the  findings  resulting  from  this  follow-on  investi¬ 
gation,  Section  2  of  this  report  describes  the  nature  of  the  cross  validation 
and  generalization  samples.  Section  3  presents  the  content  analysis  methodolo¬ 
gy  that  was  used  in  both  the  pilot  study  and  the  second  study.  Section  4  dis¬ 
cusses  the  statistical  methodology  that  was  used  to  analyze  the  data  and  the 
results  that  were  obtained.  Section  5  describes  the  design  of  the  reliability 
study  and  presents  the  results  achieved.  In  Section  6  future  areas  of  inves¬ 
tigation  are  delineated. 
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SECTION  2.  NATURE  OF  THE  PILOT  STUDY  SAMPLE  AND  THE  CROSS  VALIDATION 
AND  GENERALIZATION  SAMPLES 

As  a  result  of  research  conducted  at  the  Navy  Personnel  Research  and  De¬ 
velopment  Center,  San  Diego,  to  develop  experimental  forms  for  evaluating  per¬ 
sonnel  in  pay  grades  E-7  (Chief  Petty  Officer),  E-8  (Senior  Chief  Petty  Offi¬ 
cer),  and  E-9  (Master  Chief  Petty  Officer),  a  new  evaluation  report  form - 

NAVPERS  1616/8 - was  introduced  into  operational  use  in  January  1969  (see  Fig¬ 

ure  1),*  This  form  had  been  demonstrated  as  effecting  a  substantial  improve¬ 
ment  in  the  distribution  of  operational  evaluation  marks  over  the  previously 
used  form,  NAVPERS  792,  as  reflected  in  a  reduced  pile-up  of  marks  at  the  high 
end  of  the  marking  scale  and  greater  differentiation  among  ratees.1 

Section  19,  Evaluation  Section,  of  Evaluation  Report  Form  NAVPERS  1616/8 
is  designed  to  permit  the  rater  (evaluator)  to  compare  the  ratee  with  all 
others  of  his  rate  known  to  the  rater  on  13  specific  aspects  of  on- job  perform¬ 
ance.  Ratings  are  made  by  marking  the  column  of  the  rating  distribution  into 
which  the  rater  evaluates  that  the  ratee  falls  for  each  of  the  13  specific  as¬ 
pects  of  on- job  performance  plus  an  overall  evaluation  of  the  ratee  (for  exam¬ 
ple,  top  1%  for  superlative  performance).  Section  19R  of  this  form  provides 
space  for  the  rater  to  write  narrative  evaluation  comments  to  describe  further 
the  ratee1 s  performance  and  qualifications.  Section  19S  of  this  form  provides 
space  for  the  rater  to  write  narrative  justification  comments  and  is  required 
to  support  any  marks  assigned  to  the  top  or  bottom  10,  5,  or  1%  columns  of  Sec¬ 
tion  19. 

Sections  19R  and  19S  are  referred  to  as  the  narrative  text  of  the  Evalua¬ 
tion  Report  since  they  are  the  only  portions  of  the  report  where  the  rater 
uses  his  own  words  to  assess  the  on- job  performance  of  the  senior  enlisted  man 
that  he  is  rating.  Thus  far  the  narrative  evaluation  and  justification  sec¬ 
tions  of  the  Evaluation  Report  have  not  been  exploited  systematically  in  mak¬ 
ing  personnel  decisions  because  narrative  text  tends  to  resist  objective  analy¬ 
sis  and  interpretation. 

In  the  earlier  pilot  investigation,  the  Navy  Personnel  Research  and  De¬ 
velopment  Center  (NPRDC),  San  Diego,  selected  a  sample  of  225  Evaluation  Re¬ 
ports  for  senior  enlisted  personnel  in  pay  grade  E-7  taken  from  a  pool  of  ap¬ 
proximately  1,000  performance  evaluation  report  forms  for  two  occupational 

ratings - Aviation  Electronics  Technician  (AT)  and  Boilerman  (BT) .  All  225 

Evaluation  Reports  were  drawn  from  the  top  half  of  the  marking  scale  on  19A- 
PERFORMANCE  OF  DUTY  located  in  the  upper  right  quadrant  of  Evaluation  Report 
Form  NAVPERS  1616/8.  The  19A-PERF0RMANCE  OF  DUTY  category  was  used  in  pref¬ 
erence  to  19N-0VERALL  EVALUATION  because  standard  scores  (T  Scores)  were  avail¬ 
able  only  for  19A.  The  use  of  standard  scores  rather  than  raw  marks  permitted 
a  more  refined  selection  to  be  made  of  the  three  criterion  groups  used  in  the 
study.  Since  raw  marks  on  19A  correlate  very  highly  with  raw  marks  on  19N,  it 
was  felt  that  little  was  sacrificed  by  not  using  the  overall  evaluation  and 
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This  form  subsequently  has  been  replaced  by  another  form  that  can  be  scanned 
by  an  optical  character  reader;  however,  the  content  of  the  two  forms  is 
essentially  the  same. 
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that  much  was  gained  by  using  the  purified  T  Scores  on  19A.  Only  those  Evalua¬ 
tion  Reports  from  commands  spreading  their  marks  and  submitting  eight  or  more 
E-7  and  E-8  reports  were  considered. 

The  pilot  study  sample  of  225  Evaluation  Reports  was  divided  equally  into 
three  criterion  groups - Upper,  Middle,  and  Lower - corresponding  to  three  con¬ 

tinuous  segments  of  the  upper  half  of  the  marking  scale  on  19A-PERF0RMANCE  OF 
DUTY.  Table  1  shows  the  range  of  raw  marks  on  19A  for  each  of  the  three  cri¬ 
terion  groups  in  the  pilot  study  sample  as  well  as  the  range  and  mean  of  T 
Scores.  These  standardized  scores  have  a  mean  of  50  and  a  standard  deviation 
of  10.  Standardization  was  accomplished  by  setting  each  unit  command  mean 
equal  to  50  and  standardizing  the  total  of  E-7  and  E-8  marks  for  each  unit 
command.  No  cases  from  the  bottom  half  of  the  marking  scale  on  19A  were  in¬ 
cluded  in  this  study  since  there  is  no  difficulty  in  differentiating  these 
cases  from  the  better  performing  personnel. 

Table  2  shows  the  distribution  of  the  225  pilot  study  Evaluation  Reports 
among  the  three  criterion  groups  for  each  of  the  two  occupational  ratings  and 
for  both  occupations  combined.  After  the  pilot  study  sample  had  been  selected 
and  analyzed,  it  was  discovered  that  one  Evaluation  Report  for  an  Aviation 
Antisubmarine  Warfare  Operator  (AW)  had  erroneously  been  coded  as  an  Aviation 
Electronics  Technician  (AT) .  This  case  was  removed  from  the  analysis  by  spe¬ 
cific  occupation  but  was  left  in  the  analysis  for  the  total  pilot  study  sample. 

In  the  second  study,  NPRDC  also  selected  the  sample  of  Evaluation  Reports 
to  be  analyzed.  The  same  general  procedures  described  above  for  selecting  the 
pilot  study  sample  were  followed  also  in  selecting  the  cross  validation  sample 
and  the  generalization  sample,  except  that  the  forms  were  selected  from  a 


TABLE  1 

RANGE  OF  RAW  MARKS,  RANGE  OF  T  SCORES,  AND 
MEAN  OF  T  SCORES  ON  19A-PERF0RMANCE  OF  DUTY 
FOR  THE  THREE  CRITERION  GROUPS  IN  THE  PILOT  STUDY  SAMPLE 


Criterion 

Group 

Range  of 

Raw  Marks 

Range  of 

T  Scores 

Mean  of 

T  Scores 

Upper 

In  the  top  5% 
column  or  the  top 

1%  column 

59.3  to  74.2 

64.68 

Middle 

In  the  top  10% 
column  only 

48.0  to  54.1 

51.79 

Lower 

In  the  top  50% 
column  or  the  top 

30%  column 

33.8  to  39.7 

38.85 
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TABLE  2 

DISTRIBUTION  OF  THE  225  PILOT  STUDY  EVALUATION  REPORTS 
AMONG  THE  THREE  CRITERION  GROUPS  FOR  EACH  OF  THE  TWO 
OCCUPATIONAL  RATINGS  AND  FOR  BOTH  OCCUPATIONS  COMBINED 


Occupational 

Rating 

Criterion  Group 

Total 

N 

Upper 

Middle 

Lower 

AT 

49 

39 

56 

144 

* 

AW 

0 

1 

0 

1 

BT 

26 

35 

19 

80 

Total  Sample 

75 

75 

75 

225 

This  case  erroneously  was  coded  as  an  AT  initially. 


subsequent  year’s  data  pool.  The  cross  validation  sample  consisted  of  222 
Evaluation  Reports  from  the  same  two  ratings  that  were  used  in  the  pilot  study, 
that  is,  Aviation  Electronics  Technician  (AT)  and  Boilerman  (BT) .  In  addition, 
a  generalization  sample  consisting  of  222  Evaluation  Reports  was  selected  by 
NPRDC  from  two  different  ratings  in  order  to  ascertain  the  generalizability  of 
the  content  analytic  methodology  developed  in  the  pilot  study.  The  two  rat¬ 
ings  from  which  the  generalization  sample  was  drawn  were  Corami ssaryman  (CS) 
and  Radioman  (RM) . 

The  cross  validation  sample  of  222  Evaluation  Reports  and  the  generaliza- 
zation  sample  of  222  Evaluation  Reports  were  both  divided  equally  into  the 

same  three  criterion  groups  as  the  pilot  study  sample - Upper,  Middle,  and 

Lower.  Table  3  shows  the  range  of  raw  marks  on  19A  for  each  of  the  three  cri¬ 
terion  groups  in  the  cross  validation  sample  (AT’s  and  BTfs),  the  range  of  T 
Scores,  and  the  mean  of  the  T  Scores  for  each  criterion  group.  These  same 
data  for  the  generalization  sample  (CS’s  and  RMfs)  are  presented  in  Table  A. 
Tables  5  and  6  show  the  distribution  of  the  222  cross  validation  sample  Evalua¬ 
tion  Reports  and  the  222  generalization  sample  Evaluation  Reports  among  the 
three  criterion  groups  for  each  of  the  two  occupational  ratings  represented  in 
each  sample  and  for  both  occupations  combined.  Actual  criterion  group  member¬ 
ship  for  the  cross  validation  sample  and  the  generalization  sample  was  known 
only  to  NPRDC  until  the  content  analysis  of  the  narrative  text  had  been  com¬ 
pleted.  Consequently,  the  content  analysis  of  these  two  samples  was  conducted 
in  the  blind  without  benefit  of  knowing  to  which  criterion  group  each  Evalua¬ 
tion  Report  belonged. 
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TABLE  3 


RANGE  OF  RAW  MARKS,  RANGE  OF  T  SCORES,  AND 
MEAN  OF  T  SCORES  ON  19A-PERFORMANCE  OF  DUTY 
FOR  THE  THREE  CRITERION  GROUPS  IN  THE  CROSS  VALIDATION  SAMPLE 


Criterion 

Group 

Range  of 

Raw  Marks 

Range  of 

T  Scores 

Mean  of 

T  Scores 

Upper 

In  the  top  5% 
column  or  the  top 

1%  column 

61.2  to  71.9 

64.23 

Middle 

In  the  top  10% 
column  only 

48.2  to  55.9 

52.54 

Lower 

In  the  top  50% 
column  or  the  top 

30%  column 

30.3  to  42.0 

38.48 

TABLE  4 

RANGE  OF  RAW  MARKS,  RANGE  OF  T  SCORES,  AND 
MEAN  OF  T  SCORES  ON  19A-PERFORMANCE  OF  DUTY 
FOR  THE  THREE  CRITERION  GROUPS  IN  THE  GENERALIZATION  SAMPLE 


Criterion 

Group 

Range  of 

Raw  Marks 

Range  of 

T  Scores 

Mean  of 

T  Scores 

Upper 

In  the  top  5% 
column  or  the  top 

1%  column 

61.2  to  74.8 

64.33 

Middle 

In  the  top  10% 
column  only 

48.2  to  56.2 

52.50 

Lower 

In  the  top  50% 
column  or  the  top 

30%  column 

34.5  to  41.5 

38.56 
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TABLE  5 

DISTRIBUTION  OF  THE  222  CROSS  VALIDATION  SAMPLE 
EVALUATION  REPORTS  AMONG  THE  THREE  CRITERION  GROUPS 
FOR  EACH  OF  THE  TWO  OCCUPATIONAL  RATINGS 
AND  FOR  BOTH  OCCUPATIONS  COMBINED 


Occupational 

Rating 

Criterion  Group 

Total 

Upper 

Middle 

Lower 

N 

AT 

45 

44 

49 

138 

BT 

29 

30 

25 

84 

Total  Sample 

74 

74 

74 

222 

TABLE  6 

DISTRIBUTION  OF  THE  222  GENERALIZATION  SAMPLE 
EVALUATION  REPORTS  AMONG  THE  THREE  CRITERION  GROUPS 
FOR  EACH  OF  THE  TWO  OCCUPATIONAL  RATINGS 
AND  FOR  BOTH  OCCUPATIONS  COMBINED 


Occupational 

Rating 

Criterion  Group 

Total 

Upper 

Middle 

Lower 

N 

CS 

19 

16 

25 

60 

RM 

55 

58 

49 

162 

Total  Sample 

74 

74 

74 

222 
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SECTION  3.  CONTENT  ANALYSIS  METHODOLOGY 


Conceptual  Approach 

In  the  pilot  study,  the  narrative  portions  of  the  75  Evaluation  Reports 
for  each  of  the  three  criterion  groups  were  read  in  their  entirety  before  for¬ 
malizing  the  method  of  content  analysis  to  be  used.  In  this  review  the  evalua¬ 
tion  section  and  the  justification  section  (19R  and  19S)  were  considered  sepa¬ 
rately.  Borrowing  from  the  field  of  information  science,  it  seemed  most  ap¬ 
propriate  to  regard  each  narrative  section  as  a  short  document  that  had  been 
written  by  the  ratee fs  senior  officer  in  order  to  communicate  to  a  selection 
board  or  to  a  detailer  the  potential  that  the  ratee  had  for  promotion  and  in¬ 
creased  responsibility.  Considered  in  this  framework,  the  analysis  task  then 
becomes  one  of  ascertaining  what  the  document  is  about  (content  analysis), 
specification  of  the  content  by  a  set  of  descriptive  labels  (indexing) ,  and 
organization  of  an  indexing  vocabulary  (controlling  the  form  and  semantics  of 
the  descriptive  labels  by  lexicon  and/or  rule).3,i+  In  order  for  the  content 
analysis  to  be  valid,  Fairthorne5  cautions  that  two  aspects  must  be  taken  into 
consideration:  (a)  what  the  document  is  about,  and  (b)  the  circumstances  of 

the  expected  uses  of  the  content  analysis  with  respect  to  a  particular  task  or 
problem.  Fairthorne1 s  advice  was  attended  to  in  the  design  of  the  content 
analysis  methodology  in  that  the  indexing  vocabulary  which  was  developed  re¬ 
lates  strongly  to  the  ultimate  use  to  which  performance  evaluations  are  put, 
that  is,  the  selection  for  promotion  of  outstanding  chief  petty  officers  in 
the  face  of  limited  promotional  opportunities. 


The  Indexing  Vocabulary 

In  reading  the  narrative  portions  of  the  75  Evaluation  Reports  for  each 
of  the  three  criterion  groups  in  the  pilot  study  sample,  it  became  apparent 
that  the  attributes  and  characteristics  being  evaluated  for  a  ratee  related 
primarily  to  his  potential  as  a  manager  and  supervisor.  Consequently ,  five 
references  in  the  area  of  managerial  behavior  and  practice6* 9,10  were  con¬ 
sulted  as  an  aid  to  the  development  of  the  indexing  vocabulary  used  in  this 
study.  An  initial  vocabulary  containing  41  descriptive  labels  was  devised  and 
used  to  test  the  adequacy  and  manageability  of  the  indexing  method  on  20  Evalu¬ 
ation  Reports  not  included  in  the  pilot  study  sample  but  similar  to  them  in 
content.  As  a  result  of  this  experience,  the  original  set  of  41  labels  was 
condensed  into  a  more  generic  set  of  29  index  terms.  The  indexing  form,  in¬ 
corporating  the  final  vocabulary  that  was  used  in  both  the  pilot  study  and  the 
second  study,  is  shown  in  Figure  2. 

The  top  line  of  the  indexing  form  carries  fields  for  an  identifying  num¬ 
ber  for  each  ratee,  which  criterion  group  he  belongs  to  (used  only  in  the  pi¬ 
lot  study  since  criterion  data  were  withheld  in  the  second  study  until  the  in¬ 
dexing  had  been  completed),  and  whether  the  section  being  indexed  is  an  evalua¬ 
tion  section  (19R)  or  a  justification  section  (19S).  The  indexing  form  itself 
is  divided  into  three  major  parts:  MANAGEMENT  FUNCTIONS,  SKILLS  AND  ABILITIES, 
and  PRODUCTIVITY  AND  ACHIEVEMENT.  Under  each  of  these  headings  there  are  more 
detailed  terms,  providing  the  indexer  with  a  3-level  hierarchy  of  descriptive 
labels  from  which  to  choose. 
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ID  No.  _  _  Criterion  Group _ _  Section 

Index  Term 

MANAGEMENT  FUNCTIONS _ _ 

CONTROLLING _ _ 

LEADERSHIP  AND  DIRECTING _  _ 

ORGANIZATION _ _ 

PLANNING  _ _ _ _ _ 

REPRESENTATION _  _ 

STAFFING _ _ _ _ 

USE  OF  COMMUNICATION _  _ 

SKILLS  AND  ABILITIES _ 

COMMUNICATION _  _ 

CONDUCT,  INTEGRITY,  AND  PRIDE _  _ _ 

COOPERATION _ _ 

ENDURANCE _  _ 

FLEXIBILITY _  _ 

GROOMING  AND  ATTIRE _ _ 

INITIATIVE _  _ 

INTELLECTUAL  FUNCTIONING _ _ 

PROFESSIONALISM _ _ 

RELIABILITY  AND  DEPENDABILITY 

RESOURCEFULNESS _ 

RESPONSIVENESS _ 

TECHNICAL  SKILLS _ 

PRODUCTIVITY  AND  ACHIEVEMENT 

AWARDS  AND  PUNISHMENT _ 

DRIVE _ 

SERVICE  MOTIVATION 

POTENTIAL _ 

REPUTE _ 

ASSET  TO  THE  NAVY_ 

FREQUENCY  COUNTS:  5  _ ,  A  _ 

TOTAL  NUMBER  OF  WORDS  _ 


3  _ ,  2  _ ,  1 

TOTAL  NUMBER  OF  INDEX  TERMS 


Figure  2.  Indexing  Form  Used  in  Performing  the  Content  Analysis 
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The  first  section  of  the  indexing  form  includes  seven  specific  MANAGEMENT 
FUNCTIONS  that  many  authorities  on  management  practice  agree  are  the  character¬ 
istic  duties  of  all  managers . ® 9 ^ 9 1 ®  Although  some  authorities  believe  that 
there  are  more,  less,  or  different  functions  performed  by  managers,  these  seven 
functions  were  selected  because  they  are  representative  of  the  duties  that 
chief  petty  officers  actually  perform. 

The  second  section  of  the  indexing  form  contains  index  terms  for  13  spe¬ 
cific  SKILLS  AND  ABILITIES  considered  to  be  important  by  Navy  supervisory  per¬ 
sonnel  in  performing  effectively  as  a  chief  petty  officer.  While  some  authori¬ 
ties  on  management  practice  consider  making  a  judgment  about  whether  or  not  an 
individual  possesses  a  skill,  quality,  or  ability  to  be  a  subjective  process. 
Navy  evaluators  do  repeatedly  call  out  these  specific  qualities  in  their  narra¬ 
tive  evaluations  because  many  of  these  qualities  are  dimensions  on  which  they 
rate  the  ratee  in  Section  19  of  the  Evaluation  Report.  The  first  section  of 

the  indexing  form - MANAGEMENT  FUNCTIONS - deals  with  how  a  ratee  performs  his 

managerial  functions  and  is  result  oriented,  while  the  second  section - SKILLS 

AND  ABILITIES - contains  index  terms  that  relate  to  an  individual’s  character¬ 

istics  and  qualities  which,  if  used,  may  help  him  achieve  good  results. 

The  third  section  of  the  indexing  form— PRODUCTIVITY  AND  ACHIEVEMENT - 

is  the  most  result-oriented  section  of  the  indexing  hierarchy.  Here  are  in¬ 
cluded  the  measures  of  overall  performance.  DRIVE  and  SERVICE  MOTIVATION  (a 
specific  type  of  drive)  are  included  in  this  section  since  drive  is  considered 
to  be  one  of  the  more  important  variables  leading  to  success.  POTENTIAL  also 
is  included  here  since  potential  is  a  measure  of  future  performance.  AWARDS 
AND  PUNISHMENT,  REPUTE,  and  ASSET  TO  THE  NAVY  represent  acknowledgments  of  an 
individual’s  performance,  either  positive  or  negative  acknowledgment. 

Each  sentence  of  narrative  text  in  the  pilot  study  sample  and  the  cross 
validation  and  generalization  samples  was  read  carefully  and,  where  appropri¬ 
ate,  divided  into  segments  corresponding  to  the  assignment  of  specific  index 
terms.  However,  it  is  not  enough  to  simply  label  a  narrative  statement  with 
the  most  appropriate  index  term  since  the  statement  may  have  been  a  highly 
positive,  quite  positive,  neutral,  quite  negative,  or  highly  negative  one. 

For  example,  in  order  to  differentiate  between  the  ratee  who  plans  superbly 
and  the  ratee  who  plans  inadequately,  a  weighting  scale  was  devised  to  be 
applied  to  each  index  term  that  is  used  (see  Table  7).  The  weighting  scale 
contains  five  numerical  values  ranging  from  5  (the  positive  end  of  the  scale) 
to  1  (the  negative  end  of  the  scale).  Under  each  numerical  value  in  Table  7 
there  are  listed  samples  of  adjectives  or  adverbs  that  may  be  used  by  the 
rater  to  describe  a  ratee ’s  performance.  These  lists  of  words  provide  clues 
to  the  indexer  as  to  which  numerical  value  to  assign  to  an  index  term.  As  a 
simple  example,  if  the  rater  commented  that  the  ratee  was  highly  cooperative, 
this  statement  would  be  indexed  as  COOPERATION  and  assigned  a  weight  of  4  since 
highly  is  listed  as  an  example  under  numeral  4  in  Table  7. 

The  weighting  scale  developed  for  this  content  analysis  research  bears  a 
marked  resemblance  to  the  quality  rating  scale  developed  by  Harrington  in  rat¬ 
ing  narrative  statements  contained  in  letters  of  recommendation  regarding  can¬ 
didates  for  secondary  level  teaching  positions.11  Harrington’s  quality  rating 
scale  was  based  upon  the  proposition  that  the  favorableness  of  the  modifying 
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TABLE  7 

WEIGHTING  SCALE 


5 

4 

3 

2 

i 

excellent 

good 

average 

poor 

poorest 

superlative 

comparative 

comparative 

superlative 

best 

better  than 

average 

not  as  good 

worst 

most 

EXAMPLES 

as  most 

above 

above  average  adequate 

declining 

bottom 

reproach 

better 

aptly 

quality 

least 

beyond 

comendable 

capable 

deficiency 

lowest 

reproach 

complete 

competent 

detrimental 

boundless 

deep 

generally 

fair 

exceptional 

definitely 

moderate 

in  need  of 

extra- 

easily 

satisfac- 

insufficient 

ordinary 

effective 

tory 

lack  of 

extremely 

efficient 

sufficient- 

lower  than 

finest 

eloquent 

iy 

average 

flawless 

eminent 

usually 

lowering  of 

greatest 

exceeds 

negatively 

highest 

excels 

spotty 

ideal 

exemplary 

unfortunate 

little  to  be 

expeditious 

unwisely 

desired 

experienced 

weak  in 

limitless 

expertise 

with  the  ex¬ 

maximum 

extensive 

ception  of 

most 

favorable 

never 

great 

outstanding 

high/highly 

paramount 

immaculate 

perfect 

profound 

immensely 

impeccable 

4-good  (Cont.) 

sterling 

impressive 

rare 

superb 

innate 

remarkable 

superior 

insp ires 

significantly 

surpassed  by 

instills 

skillful 

none 

invaluable 

smoothly 

top/ topnotch 

keen 

solid 

unimpeachable 

laudable 

strongly 

unique 

leading 

surpassed 

unlimited 

marked 

thorough 

unmatched 

meticulously 

tremendous 

utmost 

model 

truly 

without  equal 

much 

unstinting 

without 

noteworthy 

valuable 

exception 

particularly 

vast 

100% 

rapidly 

very 

NOTE:  AWARDS 

AND  PUNISHMENT 

is  assigned  a  weight 

of  either  5  or 

1. 
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terms  used  by  the  writer  of  a  recommendation  is  an  index  of  his  enthusiasm  in 
recommending  the  candidate  and,  therefore,  of  the  quality  of  the  recommenda¬ 
tion.  The  rationale  for  the  weighting  scale  used  in  this  study  was  similar  to 
Harrington’s  proposition.  A  long  list  of  adjectives  and  adverbs  found  to 
occur  in  a  large  corpus  of  Evaluation  Reports  were  classified  into  five  de¬ 
grees  of  favorableness/unfavorableness  in  modifying  terms.  There  is  a  great 
deal  of  overlap  between  the  adjectives  and  adverbs  contained  in  Harrington’s 
quality  rating  scale  and  those  contained  in  the  weighting  scale  used  in  this 
research  shown  in  Table  7.  There  also  is  a  high  degree  of  correspondence  in 
the  relative  positioning  of  the  adjectives  and  adverbs  along  the  two  scales. 
The  existence  of  the  fairly  ancient  Harrington  monograph  was  not  discovered 
until  after  the  weighting  scale  used  in  this  research  had  been  devised  and  the 
research  being  reported  had  been  completed.  It  is  interesting  that  two  re¬ 
search  efforts  conducted  30  years  apart  each  independently  developed  a  similar 
conceptual  framework  for  performing  a  content  analysis  of  narrative  recommen¬ 
dations  or  evaluations.  Harrington’s  procedure  was  identical  to  the  one  fol¬ 
lowed  in  the  research  being  reported  here.  The  narrative  text  of  each  Evalua¬ 
tion  Report  was  read,  segmented  into  distinct  statements,  and  each  statement 
was  then  assigned  one  or  more  index  terms  from  the  set  of  29  possible  choices 
shown  in  Figure  2.  Each  term  selected  was  also  assigned  a  numerical  weight 
from  1  to  5  depending  upon  the  nature  of  the  adjectives  or  adverbs  used  as 
modifiers  in  the  statement.  The  following  examples  will  make  more  explicit 
the  indexing  procedure  that  was  followed. 

Example  1.  ”BTC  has  an  excellent  working  and  practical  knowledge  of  the 
PMS  System/but  has  a  tendency  to  be  lax  in  the  administrative  phase  of  the 
system. ” 

This  sentence  was  segmented  into  two  parts.  The  first  part  was  indexed 
as  TECHNICAL  SKILLS  and  assigned  a  weight  of  5.  The  second  part  was  indexed 
as  MANAGEMENT  FUNCTIONS  and  assigned  a  weight  of  2. 

Example  2.  ’’Chief  XX  was  relieved  of  his  duties  as  the  ship’s  Oil  King 
after  serving  in  the  capacity  for  approximately  two  months./  He  was  removed 
from  this  billet  because  of  his  lack  of  professional  knowledge/and  technical 
know-how  in  the  art  of  refueling.” 

This  portion  of  narrative  text  was  divided  into  three  segments  for  index¬ 
ing  purposes.  Segment  1  was  indexed  as  AWARDS  AND  PUNISHMENT  and  assigned  a 
weight  of  1.  Segment  2  was  indexed  as  PROFESSIONALISM  and  assigned  a  weight 
of  2.  Segment  3  was  indexed  as  TECHNICAL  SKILLS  and  assigned  a  weight  of  2. 

Example  3.  ”He  is  able  to  direct  the  efforts  of  Line  Personnel  in  an 
efficient  and  effective  manner; /this  is  reflected  in  the  ratee  by  a  multiple 
of  exceptional  qualities.” 

This  sentence  was  segmented  into  two  parts.  The  first  part  was  indexed 
as  LEADERSHIP  AND  DIRECTING  and  assigned  a  weight  of  4.  The  second  part  was 
indexed  as  SKILLS  AND  ABILITIES  and  assigned  a  weight  of  5. 

Example  4.  ”His  natural  abilities/and  responsible  approach  to  recruit¬ 
ing/have  enabled  the  ratee  to  outperform  his  contemporaries.” 


14 


This  sentence  was  segmented  into  three  parts.  The  first  part  was  index¬ 
ed  as  SKILLS  AND  ABILITIES  and  assigned  a  weight  of  3.  The  second  part  was 
indexed  as  RELIABILITY  AND  DEPENDABILITY  and  assigned  a  weight  of  3*  The 
third  part  was  indexed  as  PRODUCTIVITY  AND  ACHIEVEMENT  and  assigned  a  weight 
of  4 . 


Figure  3  shows  an  example  of  the  complete  narrative  text  written  in  an 
evaluation  section.  The  index  terms  that  were  selected  by  the  indexer  have 
been  recorded  above  each  segment  of  text  and  the  indexing  weights  that  were 
assigned  appear  directly  after  each  term.  Factual  statements  requiring  no  in¬ 
dexing  were  enclosed  in  brackets.  The  number  of  words  in  the  narrative  text 
were  counted  and  recorded  at  the  bottom  of  the  text  by  the  indexer. 

After  all  of  the  narrative  text  for  either  an  evaluation  section  or  a 
justification  section  of  an  Evaluation  Report  was  indexed,  the  weights  corre¬ 
sponding  to  each  term  were  written  onto  the  indexing  form  to  the  right  of  the 
appropriate  index  term  (see  Figure  4).  Thus  there  may  have  been  two  instances 
of  mention  of  the  rateefs  INTELLECTUAL  FUNCTIONING,  the  first  mention  given  a 
weight  of  3  and  the  second  a  weight  of  4.  To  the  right  of  INTELLECTUAL  FUNC¬ 
TIONING  on  the  indexing  form  for  this  ratee  would  be  written  the  following 
string  of  weights:  3,4.  Then  to  the  far  right  on  the  indexing  form  under  the 
column  headed  "Freq."  would  be  written  ,I2M,  indicating  that  this  index  term 
had  been  used  two  times  in  indexing  that  particular  section  of  narrative  text. 

At  the  bottom  of  the  indexing  form  there  is  a  line  labeled  FREQUENCY 
COUNTS.  After  all  of  the  weights  assigned  to  the  index  terms  selected  for  a 
section  of  narrative  text  (19R  or  19S)  had  been  entered  on  the  indexing  form, 
all  of  the  5  weights  were  counted  and  the  sum  was  entered  to  the  right  of  5  on 
the  FREQUENCY  COUNTS  line.  The  same  procedure  was  followed  for  entering  the 
frequency  count  of  4  weights,  3  weights,  2  weights,  and  1  weights.  The  final 
step  in  completing  the  indexing  form  was  to  transfer  the  total  number  of  words 
written  at  the  bottom  of  the  narrative  text  and  to  count  the  total  number  of 
index  terms  selected  from  the  set  of  29  possibilities. 

In  order  to  increase  the  likelihood  of  consistent  usage  of  the  indexing 
vocabulary,  a  definition  was  written  for  each  of  the  29  index  terms.  Koontz 
and  O'Donnell's  Principles  of  Management 7  was  relied  upon  heavily  in  defining 
the  management-oriented  terms  listed  in  Figure  2.  Also  contributing  to  the 
formulation  of  the  definitions  for  the  29  index  terms  was  the  way  that  Navy 
evaluators  actually  referred  to  these  concepts  in  narrative  text.  These  defi¬ 
nitions  were  consulted  frequently  during  the  indexing  process.  Indexing  of 
the  pilot  study  sample  and  the  cross  validation  and  generalization  samples  was 
performed  by  one  experienced  indexer  who  also  had  conceptualized  the  content 
of  the  indexing  vocabulary  and  had  prepared  the  definitions  of  the  29  terms. 

As  part  of  the  concomitant  study  to  ascertain  the  reliability  of  this  content 
analysis  methodology,  a  training  manual  was  developed  for  use  by  the  four  re¬ 
liability  indexers  participating  in  the  study.  This  training  manual  is  in¬ 
cluded  in  its  entirety  in  Appendix  A  and  incorporates  an  alphabetical  diction¬ 
ary  of  the  29  index  terms.  The  dictionary  definition  for  each  term  is  follow¬ 
ed  by  extensive  examples  of  correct  indexing  usage  of  the  term  and  the  proper 
assignment  of  weights. 


15 


As  Baxendale  has  so  cogently  articulated,  "The  core  problem  of  content 
analysis  concerns  language  in  both  its  connotative  and  denotative  aspects — 
a  Pandora’s  box  of  semantic  and  psychological  complexities  and  unknowns,"12 
Acknowledging  these  obstacles  to  a  perfect  representation  of  the  content  of  a 
particular  segment  of  narrative  text,  numerous  examples  of  indexing  usage  have 
been  included  in  the  alphabetical  dictionary  of  index  terms,  primarily  as  a 
means  of  illustrating  the  level  of  objectivity  and  consistency  that  can  be 
achieved  in  this  type  of  content  analysis,  but  also  to  demonstrate  the  intrac¬ 
table  problems  that  still  reside  in  any  effort  to  organize  and  objectify  the 
domain  of  linguistic  discourse. 
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Ratee  is  an  intelligent  and  proficient  Petty  Officer,  who  performs  his  duties 

*****  ©x$3  L*P3 

in  an  outstanding  manner.  His  ability  to  plan,  organize,  coordinate  and  super- 

+  ACH  3 

vise  have  been  ably  demonstrated  by  his  performance  as  Recruit  Company  Command- 

*€L  >  p#/  3  ML  v>  >#/•  j  ^ 

er.  Ratee  is  dependable,  trustworthy,  and  exhibits  mature  judgment  in  dispos¬ 
er  rtf*  ¥ 

ing  of  problems  which  occur  within  his  company.  Ratee’ s  military  appearance 
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and  neatness  of  person  and  dress  denote  great  pride.  He  is  cheerful,  highly 

va iv  v  r 

motivated,  and  gets  along  exceptionally  well  with  others.  Ratee fs  command  of 

<#M/H  ♦ 

the  English  language,  both  orally  and  written  is  above  average.  Ratee  is  high- 

fT  H  /«• 

ly  recommended  for  E-8.  I  Ratee  has  been  in  Water  Survival  and  Hygiene  Division 

n  fOT  ¥ 

only  for  a  short  period  of  time.]  He  has  shown  a  great  potential  towards  being 

tie  p*iv  3 

a  swimming  instructor.  Ratee  is  practicing  on  his  own  time  to  qualify  for 


Senior  Life  Saver. 

T  *  /35- 


Figure  3.  Example  of  the  Narrative  Text  for  An  Evaluation  Section  Show¬ 
ing  the  Indexing  Decisions  That  Were  Made.  Factual  Statements 
Requiring  No  Indexing  Are  Enclosed  in  Brackets.  T  =  Total 
Number  of  Words  in  the  Narrative  Text. 
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ID  No.  IOOO _  Criterion  Group  1  *4.  Section  (lift) 

~trj- 

Index  Term  Freq . 


MANAGEMENT  FUNCTIONS 

4 

1 

CONTROLLING 

LEADERSHIP  AND  DIRECTING 

3 

1 

ORGANIZATION 

3.  3 

2 

PLANNING 

/ 

3  

i 

REPRESENTATION 

STAFFING 

USE  OF  COMMUNICATION 


SKILLS  AND  ABILITIES 


COMMUNICATION 

4 

1 

CONDUCT,  INTEGRITY,  AND  PRIDE 

1  1 

2 

COOPERATION 

7 - 

r 

1 

ENDURANCE 

FLEXIBILITY 

GROOMING  AND  ATTIRE 

3 

1 

INITIATIVE 

INTELLECTUAL  FUNCTIONING 

3  4 

2 

PROFESSIONALISM 

7 

RELIABILITY  AND  DEPENDABILITY 

CO 

l  1 

2 

RESOURCEFULNESS 

7 - 

RESPONSIVENESS 

TECHNICAL  SKILLS 

PRODUCTIVITY  AND  ACHIEVEMENT 

1  1 

2 

AWARDS  AND  PUNISHMENT 

7 - 

DRIVE 

4  . 3 

2 

SERVICE  MOTIVATION 

r 

POTENTIAL 

\  i 

2 

REPUTE 

- 7 - 

ASSET  TO  THE  NAVY 

FREQUENCY  COUNTS:  52  ,  4  7  ,3 

II  ,  2 

,  1 

TOTAL  NUMBER  OF  WORDS  j3_£_  TOTAL  NUMBER  OF  INDEX  TERMS  /  3  _ 

Figure  4.  The  Indexing  Form  As  It  Was  Filled  Out  to  Record  the  Indexing  De¬ 
cisions  Made  in  the  Example  of  Narrative  Text  Shown  in  Figure  3 
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SECTION  A.  STATISTICAL  ANALYSIS  AND  RESULTS 

A  set  of  67  quantitative  variables  was  derived  from  the  indexing  form 
used  in  the  content  analysis  (see  Table  8).  The  first  29  variables  reflect 
the  simple  frequency  with  which  each  index  term  was  used  to  index  a  particular 
section  of  narrative  text.  Variable  30  is  the  sum  of  these  29  frequencies. 
Variables  31  through  59  represent  the  weighted  frequency  of  each  index  term 
used  to  index  a  particular  section  of  narrative  text.  For  example,  suppose 
that  the  index  term  CONTROLLING  was  used  twice.  The  first  time  that  it  was 
used  it  was  assigned  a  weight  of  A;  the  second  time  that  it  was  used  it  was 
assigned  a  weight  of  3.  The  weighted  frequency  then  for  CONTROLLING  would  be 
Axl  +  3xl  «  7.  The  simple  frequency  for  this  same  example  would  be  1  +  1  ■  2, 
Variable  60  is  similar  to  Variable  30  in  that  it  is  the  sum  of  the  29  weighted 
frequencies. 

Variables  61  through  65  represent  the  frequency  counts  over  the  entire 
indexing  form  for  all  5  weights,  A  weights,  3  weights,  2  weights,  and  1 
weights.  Variable  66  is  the  total  number  of  words  in  the  section  of  narrative 
text  that  was  indexed.  Variable  67  is  the  total  number  of  index  terms  of  the 
29  available  that  were  used  to  index  the  section  of  narrative  text. 

Profiles  or  vectors  of  these  67  values  then  were  prepared  for  all  of  the 
Evaluation  Reports  contained  in  each  sample.  Separate  profiles  were  compiled 
for  the  evaluation  and  justification  sections  of  each  Evaluation  Report.  If 
certain  index  terms  were  not  used  at  all  in  indexing  the  evaluation  section 
narrative  or  the  justification  section  narrative,  they  were  given  a  value  of 
zero  in  the  profile.  This  practice  raised  an  important  theoretical  issue.  Is 
it  more  damaging  not  to  say  anything  about  a  ratee’s  performance  in  a  particu¬ 
lar  area  than  to  damn  him  with  qualified  praise?  A  statement  such  as  the  fol¬ 
lowing  was  assigned  a  weight  of  2:  "With  more  time  and  conscientious  effort, 
he  should  realize  a  greater  potential."  This  evaluation  of  the  ratee’s  poten¬ 
tial  seems  more  negative  than  not  to  have  commented  at  all  about  his  potential. 

As  a  result  of  these  considerations,  the  weighting  scale  that  had  been 
used  in  the  indexing  of  Variables  31  through  59  was  transformed  in  order  to 
place  no  comment  between  positive  comments  and  negative  comments.  Table  9 
shows  the  conversion  that  was  used.  A  constant  of  10  was  added  to  the  weight¬ 
ed  frequency  of  Variables  31  through  59  in  order  to  avoid  the  incidence  of  any 
negative  input  values  in  the  subsequent  statistical  computations. 

All  profiles  were  transformed  to  the  new  weighting  scale  and  entered  onto 
IBM  coding  forms  in  preparation  for  keypunching.  The  criterion  data  and  occu¬ 
pational  rating  codes  were  known  for  the  pilot  study  sample  and  were  included 
on  the  coding  forms.  However,  all  of  the  coding  forms  for  the  cross  valida¬ 
tion  sample  and  the  generalization  sample  were  sent  to  the  Navy  Personnel  Re¬ 
search  and  Development  Center  in  San  Diego  where  the  criterion  data  and  occu¬ 
pational  rating  codes  were  added  to  the  coding  forms  and  then  returned  to  R-K 
Research  and  System  Design  for  keypunching  at  UCLA.  Card  decks  for  each  of 
the  three  samples  were  assembled  in  six  parts:  (1)  Upper  Criterion  Group  - 
Evaluation  Section,  (2)  Middle  Criterion  Group  -  Evaluation  Section,  (3)  Lower 
Criterion  Group  -  Evaluation  Section,  (A)  Upper  Criterion  Group  -  Justifica¬ 
tion  Section,  (5)  Middle  Criterion  Group  -  Justification  Section,  and  (6) 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 


TABLE  8 


DEFINITION  OF  THE  67  QUANTITATIVE  VARIABLES 
DERIVED  FROM  THE  INDEXING  FORM 


Description  of  Variable 


Frequency  of  Mention  of  MANAGEMENT  FUNCTIONS 

Frequency  of  Mention  of  CONTROLLING 

Frequency  of  Mention  of  LEADERSHIP  AND  DIRECTING 

Frequency  of  Mention  of  ORGANIZATION 

Frequency  of  Mention  of  PLANNING 

Frequency  of  Mention  of  REPRESENTATION 

Frequency  of  Mention  of  STAFFING 

Frequency  of  Mention  of  USE  OF  COMMUNICATION 

Frequency  of  Mention  of  SKILLS  AND  ABILITIES 

Frequency  of  Mention  of  COMMUNICATION 

Frequency  of  Mention  of  CONDUCT,  INTEGRITY,  AND 

PRIDE 

Frequency  of  Mention  of  COOPERATION 
Frequency  of  Mention  of  ENDURANCE 
Frequency  of  Mention  of  FLEXIBILITY 
Frequency  of  Mention  of  GROOMING  AND  ATTIRE 
Frequency  of  Mention  of  INITIATIVE 
Frequency  of  Mention  of  INTELLECTUAL  FUNCTIONING 
Frequency  of  Mention  of  PROFESSIONALISM 
Frequency  of  Mention  of  RELIABILITY  AND  DEPENDA¬ 
BILITY 

Frequency  of  Mention  of  RESOURCEFULNESS 
Frequency  of  Mention  of  RESPONSIVENESS 
Frequency  of  Mention  of  TECHNICAL  SKILLS 
Frequency  of  Mention  of  PRODUCTIVITY  AND 

ACHIEVEMENT 

Frequency  of  Mention  of  AWARDS  AND  PUNISHMENT 

Frequency  of  Mention  of  DRIVE 

Frequency  of  Mention  of  SERVICE  MOTIVATION 

Frequency  of  Mention  of  POTENTIAL 

Frequency  of  Mention  of  REPUTE 

Frequency  of  Mention  of  ASSET  TO  THE  NAVY 

Sum  of  Variables  1  through  29 

Weighted  Frequency  of  Mention  of  MANAGEMENT 

FUNCTIONS 

Weighted  Frequency  of  Mention  of  CONTROLLING 
Weighted  Frequency  of  Mention  of  LEADERSHIP 

AND  DIRECTING 

Weighted  Frequency  of  Mention  of  ORGANIZATION 
Weighted  Frequency  of  Mention  of  PLANNING 
Weighted  Frequency  of  Mention  of  REPRESENTATION 


(Continued) 
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TABLE  8  (CONT , ) 

DEFINITION  OF  THE  67  QUANTITATIVE  VARIABLES 
DERIVED  FROM  THE  INDEXING  FORM 


Number  of 
Variable 


Description  of  Variable 


37 

Weighted 

Frequency 

of 

Mention  of 

STAFFING 

38 

Weighted 

Frequency 

of 

Mention  of 

USE  OF 

COMMUNICATION 

39 

Weighted 

Frequency 

of 

Mention  of 

SKILLS  AND 

ABILITIES 

40 

Weighted 

Frequency 

of 

Mention  of 

COMMUNICATION 

41 

Weighted 

Frequency 

of 

Mention  of 

CONDUCT,  INTE- 

GRITY,  AND  PRIDE 

42 

Weighted 

Frequency 

of 

Mention  of 

COOPERATION 

43 

Weighted 

Frequency 

of 

Mention  of 

ENDURANCE 

44 

Weighted 

Frequency 

of 

Mention  of 

FLEXIBILITY 

45 

Weighted 

Frequency 

of 

Mention  of 

GROOMING  AND 

ATTIRE 

46 

Weighted 

Frequency 

of 

Mention  of 

INITIATIVE 

47 

Weighted 

Frequency 

of 

Mention  of 

INTELLECTUAL 

FUNCTIONING 

48 

Weighted 

Frequency 

of 

Mention  of 

PROFESSIONALISM 

49 

Weighted 

Frequency 

of 

Mention  of 

RELIABILITY  AND 

DEPENDABILITY 

50 

Weighted 

Frequency 

of 

Mention  of 

RESOURCEFULNESS 

51 

Weighted 

Frequency 

of 

Mention  of 

RESPONSIVENESS 

52 

Weighted 

Frequency 

of 

Mention  of 

TECHNICAL  SKILLS 

53 

Weighted 

Frequency 

of 

Mention  of 

PRODUCTIVITY  AND 

ACHIEVEMENT 

54 

Weighted 

Frequency 

of 

Mention  of 

AWARDS  AND 

PUNISHMENT 

55 

Weighted 

Frequency 

of 

Mention  of 

DRIVE 

56 

Weighted 

Frequency 

of 

Mention  of 

SERVICE  MOT IV A- 

TION 

57 

Weighted 

Frequency 

of 

Mention  of 

POTENTIAL 

58 

Weighted 

Frequency 

of 

Mention  of 

REPUTE 

59 

Weighted 

Frequency 

of 

Mention  of 

ASSET  TO  THE 

60 

61 

62 

63 

64 

65 

66 
67 


NAVY 

Sum  of  Variables  31  through  59 


Total  Number  of 
Total  Number  of 
Total  Number  of 
Total  Number  of 
Total  Number  of 


5  Weights 
4  Weights 
3  Weights 
2  Weights 
1  Weights 

Total  Number  of  Words  in  Narrative  Text 
Total  Number  of  Index  Terms  Used 
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TABLE  9 

TRANSFORMATION  OF  WEIGHTING  SCALE 


Original 

Weights 

5  (Excellent) 

4  (Good) 

3  (Average) 

2  (Poor) 

1  (Poorest) 

0  (No  Comment) 


Transformed 

Weights 


3  (Excellent) 

2  (Good) 

1  (Average) 

0  (No  Comment) 
-1  (Poor) 

-2  (Poorest) 


Lower  Criterion  Group  -  Justification  Section.  Duplicate  card  decks  were  sent 
to  NPRDC. 

In  order  to  better  visualize  the  nature  of  the  distributions  of  the  67 
variables  for  each  of  the  three  research  samples,  computer  runs  were  made  at 
the  UCLA  Health  Sciences  Computing  Facility*  using  Program  BMD01D  of  the  li¬ 
brary  of  Biomedical  Computer  Programs.13  Program  BMD01D  -  Simple  Data  Descrip¬ 
tion  computes  arithmetic  means,  standard  deviations,  standard  errors  of  means, 
maximum  values,  minimum  values,  ranges,  and  sample  sizes  for  a  set  of  input 
variables.  The  output  from  these  computer  runs  for  the  cross  validation  sam¬ 
ple  and  the  generalization  sample  is  presented  in  Appendix  B.  The  output  from 
the  computer  run  for  the  pilot  study  sample  was  included  in  an  earlier  techni¬ 
cal  report,14  and  since  it  is  rather  voluminous,  it  is  not  repeated  in  this  re¬ 
port. 


Some  interesting  comparisons  can  be  made  from  the  descriptive  statistics 
compiled  in  Appendix  B.  Tables  10  and  11  present  the  data  on  Variable  66,  to¬ 
tal  number  of  words  contained  in  the  narrative  text  of  the  evaluation  and  jus¬ 
tification  sections  of  the  Evaluation  Report,  Table  10  for  the  cross  valida¬ 
tion  sample  and  Table  11  for  the  generalization  sample.  The  results  for  the 
evaluation  sections  of  Tables  10  and  11  are  different  from  those  found  in  the 
pilot  study15  where  the  nearer  that  the  criterion  group  was  to  the  top  of  the 
distribution  of  T  Scores  on  19A-PERF0RMANCE  OF  DUTY,  the  longer  the  narrative 
text  written  to  evaluate  the  performance  of  the  ratee.  In  the  pilot  study 
sample  approximately  89  words  were  written  on  the  average  to  evaluate  ratees 
in  the  Upper  criterion  group.  This  average  dropped  to  approximately  76  words 
in  the  Middle  criterion  group  and  to  approximately  67  words  in  the  Lower  cri¬ 
terion  group.  This  decreasing  trend  is  not  apparent  in  Tables  10  and  11  for 
the  evaluation  section.  Instead,  in  the  cross  validation  sample  the  trend  is 
just  the  opposite,  with  the  average  length  of  the  narrative  text  of  the  evalua¬ 
tion  section  becoming  longer  as  the  criterion  group  moves  farther  away  from 


* 


Computing  assistance  was  obtained  from  the  Health  Sciences  Computing  Facility, 
UCLA,  sponsored  by  NIH  Special  Research  Resources  Grant  RR-3. 
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TABLE  10 


DESCRIPTIVE  STATISTICS  ON  VARIABLE  66: 
TOTAL  NUMBER  OF  WORDS  IN  NARRATIVE  TEXT 
FOR  THE  CROSS  VALIDATION  SAMPLE 


Evaluation 


Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

78.3376 

49.0724 

212 

0 

212 

Middle 

86.5808 

42.8689 

214 

0 

214 

Lower 

87.1889 

46.2204 

287 

0 

287 

Justification 

Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

201.4052 

146.3721 

896 

35 

861 

Middle 

146.5944 

124.8155 

820 

17 

803 

Lower 

17.6754 

27.2311 

112 

0 

112 

TABLE  11 

DESCRIPTIVE  STATISTICS  ON  VARIABLE  66: 
TOTAL  NUMBER  OF  WORDS  IN  NARRATIVE  TEXT 
FOR  THE  GENERALIZATION  SAMPLE 


Evaluation 


Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

98.1754 

83.3072 

530 

0 

530 

Middle 

84.5132 

53.0552 

293 

0 

293 

Lower 

100.2835 

54.5660 

353 

0 

353 

rustification 

Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

228.2968 

155.7997 

881 

17 

864 

Middle 

133.0268 

96.9929 

442 

0 

442 

Lower 

27.4051 

40.0231 

172 

0 

172 
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the  top  of  the  distribution  of  T  Scores  on  19A,  In  the  generalization  sample 
the  average  length  of  the  narrative  text  of  the  evaluation  section  drops  from 
the  Upper  criterion  group  to  the  Middle  criterion  group  but  then  rises  even 
higher  for  the  Lower  criterion  group  than  for  the  Upper  criterion  group.  These 
unexpected  findings  for  the  cross  validation  and  generalization  samples  can  be 
explained  by  the  presence  of  a  few  atypically  long  evaluation  sections  in  the 
Middle  and  Lower  criterion  groups  that  markedly  influenced  the  mean  length  for 
these  two  criterion  groups.  This  explanation  is  further  corroborated  by  exam¬ 
ining  the  maximum  total  number  of  words  contained  in  the  narrative  text  of  the 
evaluation  section  for  the  Lower  criterion  group  as  shown  in  Tables  10  and  11. 

A  minimum  of  zero  words  resulted  from  evaluation  sections  not  being  written  at 
all  for  a  handful  of  cases  in  the  various  criterion  groups. 

In  the  pilot  study  the  trend  of  longer  narrative  text  for  criterion  groups 
near  the  top  of  the  distribution  of  T  Scores  on  19A-PERF0RMANCE  OF  DUTY  pre¬ 
vailed  also  for  the  justification  section,  but  the  effect  was  even  more  pro¬ 
nounced  than  for  the  evaluation  section.  This  same  outcome  resulted  for  the 
justification  section  in  the  cross  validation  and  generalization  samples,  and 
can  be  explained  partly  by  the  requirement  to  write  an  individual  justification 
in  Section  19S  for  any  mark  given  in  the  top  10,  5,  or  1%  columns  of  Section 
19.  When  a  justification  section  was  written,  on  the  average  it  was  longer 
than  the  evaluation  section  in  the  Upper  and  Middle  criterion  groups  of  all 
three  samples. 

Tables  12  and  13  also  extract  data  from  Appendix  B  and  present  descriptive 
statistics  for  all  three  criterion  groups  in  the  cross  validation  and  generali¬ 
zation  samples  on  Variable  67,  total  number  of  index  terms  used  of  the  29 
available  to  index  the  evaluation  and  justification  sections  of  the  Evaluation 
Report.  The  results  shown  in  Tables  12  and  13  are  correlated  to  those  present¬ 
ed  in  Tables  10  and  11  in  that  the  number  of  index  terms  used  is  a  function  of 
length  of  the  narrative  text.  As  the  evaluator  uses  more  words  to  describe 
the  ratee,  he  is  more  likely  to  comment  on  a  wider  variety  of  specific  areas 
of  the  ratee* s  performance.  This  tendency  is  mirrored  in  the  number  of  dif¬ 
ferent  index  terms  selected  by  the  indexer  to  encompass  the  narrative  content. 
It  is  interesting  that  on  the  average  in  the  evaluation  section  of  either  the 
cross  validation  or  the  generalization  sample  only  a  half  dozen  substantive 
areas  of  the  ratee* s  performance  were  described  of  the  29  possibilities.  This 
was  also  the  finding  in  the  pilot  study. *6  Even  in  the  justification  section 
where  longer  expositions  were  written,  on  the  average  only  11  or  12  of  the  29 
content  areas  were  mentioned  in  the  Upper  criterion  group  (nine  areas  in  the 
pilot  study  Upper  criterion  group*6).  This  finding,  consistent  across  all 
three  samples,  suggests  that  the  evaluators  may  be  victims  of  habitual  ways  of 
formulating  and  phrasing  the  narrative  sections  of  the  Evaluation  Report.  Of 
the  richness  of  information  that  could  be  used  to  describe  the  performance  of 
ratees  in  the  Upper  criterion  group,  on  the  average  only  a  partial  representa¬ 
tion  is  utilized.  All  of  the  29  index  terms  were  used  at  one  time  or  another 
to  index  the  evaluation  sections  or  the  justification  sections  of  the  three 
criterion  groups  in  all  three  samples.  Therefore,  one  can  conclude  that  al¬ 
though  only  a  partial  representation  of  the  29  substantive  areas  may  be  utiliz¬ 
ed  to  describe  a  particular  ratee,  over  a  sample  as  large  as  any  one  criterion 
group  (N=74  or  75),  all  29  areas  of  performance  do  get  mentioned  at  one  time 
or  another. 


25 


TABLE  12 

DESCRIPTIVE  STATISTICS  ON  VARIABLE  67: 
TOTAL  NUMBER  OF  INDEX  TERMS  USED 
FOR  THE  CROSS  VALIDATION  SAMPLE 


Evaluation 


Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

5.6081 

3.3508 

14 

0 

14 

Middle 

7.1621 

3.2011 

15 

0 

15 

Lower 

6.5135 

2.9759 

16 

0 

16 

Justification 

Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

11.6081 

4.8224 

27 

4 

23 

Middle 

9.1621 

4.3132 

19 

1 

18 

Lower 

1.6081 

2.4372 

9 

0 

9 

TABLE  13 

DESCRIPTIVE  .STATISTICS  ON  VARIABLE  67 : 
TOTAL  NUMBER  OF  INDEX  TERMS  USED 
FOR  THE  GENERALIZATION  SAMPLE 


Evaluation 


Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

6.3378 

4.7637 

22 

0 

22 

Middle 

5.8648 

3.3365 

14 

0 

14 

Lower 

6.3648 

2.8020 

14 

0 

14 

rustification 

Section 

Mean 

S.D. 

Max. 

Min. 

Range 

Upper 

12.4730 

4.6852 

22 

1 

21 

Middle 

8.4730 

4.2819 

18 

0 

18 

Lower 

1.8648 

2.5660 

10 

0 

10 
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From  the  duplicate  deck  of  punched  cards  for  the  cross  validation  and  the 
generalization  samples  that  was  sent  to  NPRDC,  correlation  matrices  were  com¬ 
puted  at  the  computing  facility  of  the  Naval  Electronics  Laboratory  Center  in 
San  Diego.  Correlations  among  all  67  variables  for  the  evaluation  section  and 
for  the  justification  section  were  computed  for  each  sample  as  well  as  the  ma¬ 
trix  correlating  the  evaluation  section  with  the  justification  section  on  all 
67  variables.  These  results  are  displayed  in  Appendix  C.  The  reader  is  refer¬ 
red  to  Table  8  for  a  definition  of  each  of  the  67  variables. 

The  overall  picture  that  emerges  from  a  scrutiny  of  these  six  matrices  is 
that  there  are  very  few  high  correlations  except  for  the  correlations  between 
Variables  1  through  30  and  Variables  31  through  60  which  are  shown  in  italic 
type.  These  two  sets  of  variables  are  the  same  except  for  the  method  of 
weighting  that  was  used.  The  other  variables  that  showed  a  consistently  high 
correlation  in  the  evaluation  section  and  also  in  the  justification  section 
for  both  samples  were  Variable  30  (Sum  of  Variables  1  through  29)  and  Variable 
60  (Sum  of  Variables  31  through  59)  with  Variable  61  (Total  Number  of  5  [New 
3]  Weights),  Variable  62  (Total  Number  of  4  [New  2]  Weights),  Variable  63  (To¬ 
tal  Number  of  3  [New  1]  Weights),  Variable  66  (Total  Number  of  Words  in  the 
Narrative  Text),  and  Variable  67  (Total  Number  of  Index  Terms  Used).  Variable 
67  (Total  Number  of  Index  Terms  Used)  is  a  focal  variable,  correlating  highly 
with  Variable  30  (Sum  of  Variables  1  through  29) ,  Variable  60  (Sum  of  Variables 
31  through  59),  Variable  61  (Total  Number  of  5  [New  3]  Weights),  Variable  62 
(Total  Number  of  4  [New  2]  Weights),  Variable  63  (Total  Number  of  3  [New  1] 
Weights),  and  Variable  66  (Total  Number  of  Words  in  the  Narrative  Text).  Cor¬ 
relations  among  the  variables  having  to  do  with  the  29  index  terms  per  se  were 
notably  low,  indicating  that  these  29  dimensions  are  relatively  orthogonal  and 
represent  independent  aspects  of  managerial  performance.  In  the  correlation 
matrix  for  the  evaluation  section  versus  the  justification  section,  no  high 
correlations  were  evident  in  either  the  cross  validation  sample  or  the  gener¬ 
alization  sample,  demonstrating  that  these  two  narrative  sections  of  Evalua¬ 
tion  Report  Form  NAVPERS  1616/8  are  quite  independent  and  should  be  treated 
separately.  An  early  decision  in  this  research  project  was  to  treat  these  two 
narrative  sections  separately;  in  retrospect  this  determination  appears  to 
have  been  a  wise  decision. 

Since  the  system  that  was  used  for  weighting  Variables  31  through  59  has 
been  highlighted  by  the  correlational  analysis  as  an  important  aspect  of  the 
content  analysis  methodology,  it  is  of  interest  to  see  the  distribution  of  in¬ 
dex  weights  used  for  each  of  the  three  criterion  groups  in  the  cross  valida¬ 
tion  and  generalization  samples.  These  results  are  presented  in  Tables  14 
through  17.  Tables  14  and  15  show  the  distribution  for  the  cross  validation 
sample,  first  for  the  evaluation  section  and  then  for  the  justification  sec¬ 
tion.  Comparable  distributions  for  the  generalization  sample  are  shown  in 
Tables  16  and  17.  A  chi  square  test  of  common  distribution  was  calculated  for 
all  pairwise  criterion  group  comparisons  in  Tables  14  through  17  in  order  to 
test  the  hypothesis  that  the  distributions  of  index  weights  used  for  each  pair 
of  groups  were  drawn  from  the  same  population.  This  hypothesis  was  rejected 
beyond  the  .001  level  of  probability  for  five  of  the  six  comparisons  made  for 
the  cross  validation  sample  (see  Table  18).  For  the  comparison  of  the  Upper 
versus  Middle  criterion  groups  on  the  justification  section,  the  hypothesis 
was  rejected  beyond  the  .01  level  of  probability.  These  results  replicate  the 
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TABLE  14 


DISTRIBUTION  OF  TOTAL  NUMBER  OF  5  WEIGHTS  THROUGH  1  WEIGHTS 
USED  IN  INDEXING  THE  EVALUATION  SECTION  (19R) 

OF  THE  CROSS  VALIDATION  SAMPLE  FOR  ALL  THREE  CRITERION  GROUPS 


Weights 

Criterion  Group 

Upper 

Middle 

Lower 

5 

(New  3)  Excellent 

203 

157 

84 

4 

(New  2)  Good 

270 

324 

314 

3 

(New  1)  Average 

144 

261 

234 

2 

(New  -1)  Poor 

0 

14 

71 

1 

(New  -2)  Poorest 

0 

0 

1 

TABLE  15 

DISTRIBUTION  OF  TOTAL  NUMBER  OF  5  WEIGHTS  THROUGH  1  WEIGHTS 
USED  IN  INDEXING  THE  JUSTIFICATION  SECTION  (19S) 

OF  THE  CROSS  VALIDATION  SAMPLE  FOR  ALL  THREE  CRITERION  GROUPS 


Weights 

Criterion  Group 

Upper 

Middle 

Lower 

5  (New  3)  Excellent 

508 

265 

48 

4  (New  2)  Good 

738 

486 

71 

3  (New  1)  Average 

528 

408 

38 

2  (New  -1)  Poor 

0 

2 

2 

1  (New  -2)  Poorest 

0 

0 

1 
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TABLE  16 


DISTRIBUTION  OF  TOTAL  NUMBER  OF  5  WEIGHTS  THROUGH  1  WEIGHTS 
USED  IN  INDEXING  THE  EVALUATION  SECTION  (19R) 

OF  THE  GENERALIZATION  SAMPLE  FOR  ALL  THREE  CRITERION  GROUPS 


Weights 

Criterion  Group 

Upper 

Middle 

Lower 

5  (New  3)  Excellent 

220 

137 

115 

4  (New  2)  Good 

327 

287 

317 

3  (New  1)  Average 

199 

205 

211 

2  (New  -1)  Poor 

1 

17 

54 

1  (New  -2)  Poorest 

0 

0 

1 

TABLE  17 

DISTRIBUTION  OF  TOTAL  NUMBER  OF  5  WEIGHTS  THROUGH  1  WEIGHTS 
USED  IN  INDEXING  THE  JUSTIFICATION  SECTION  (19S) 

OF  THE  GENERALIZATION  SAMPLE  FOR  ALL  THREE  CRITERION  GROUPS 


f 


Weights 

Criterion  Group 

Upper 

Middle 

Lower 

5  (New  3)  Excellent 

451 

231 

46 

4  (New  2)  Good 

783 

444 

105 

3  (New  1)  Average 

579 

385 

51 

2  (New  -1)  Poor 

5 

3 

3 

1  (New  -2)  Poorest 

0 

0 

0 

29 


TABLE  18 

RESULTS  OF  THE  CHI  SQUARE  TESTS  OF  COMMON  DISTRIBUTION 
OF  INDEX  WEIGHTS  USED  FOR  EACH  PAIR  OF  CRITERION  GROUPS 
IN  THE  CROSS  VALIDATION  SAMPLE 


Criterion  Group 

Comparison 

Value  of 
Chi  Square 

Degrees  of 
Freedom 

Probability 

Level 

Evaluation  Section 

Upper  vs.  Middle 

44.9757 

4 

P  < 

.001 

Middle  vs.  Lower 

61.1906 

4 

P  < 

.001 

Upper  vs.  Lower 

140.9668 

4 

P  < 

.001 

Justification  Section 

Upper  vs.  Middle 

18.4300 

4 

.001  < 

P  <  .1 

Middle  vs .  Lower 

21.3217 

4 

P  < 

.001 

Upper  vs .  Lower 

35.4408 

4 

P  < 

.001 

TABLE  19 

RESULTS  OF  THE  CHI  SQUARE  TESTS  OF  COMMON  DISTRIBUTION 
OF  INDEX  WEIGHTS  USED  FOR  EACH  PAIR  OF  CRITERION  GROUPS 
IN  THE  GENERALIZATION  SAMPLE 


Criterion  Group 

Comparison 

Value  of 
Chi  Square 

Degrees  of 
Freedom 

Probability 

Level 

Evaluation  Section 

Upper  vs.  Middle 

29.0438 

4 

p  < 

.001 

Middle  vs.  Lower 

21.7997 

4 

p  < 

.001 

Upper  vs.  Lower 

83.9246 

4 

p  < 

.001 

Justification  Section 

Upper  vs.  Middle 

6.7782 

4 

p  > 

.05 

Middle  vs.  Lower 

15.0797 

4 

.001  < 

P  <  .01 

Upper  vs.  Lower 

12.6617 

4 

.01  < 

P  <  .05 

Note:  x2  with  4  degrees 

•001 

of  freedom  * 

18.465. 
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findings  in  the  earlier  pilot  study.1  For  the  generalization  sample  the  hy¬ 
pothesis  was  rejected  beyond  the  .001  level  of  probability  for  all  three  com¬ 
parisons  made  for  the  evaluation  section  (see  Table  19).  However,  in  the  jus¬ 
tification  section  the  difference  between  the  distributions  of  weights  for  the 
Upper  versus  Middle  criterion  groups  was  not  statistically  significant.  The 
other  two  comparisons  (i.e.,  Middle  vs.  Lower  and  Upper  vs.  Lower)  were  sta¬ 
tistically  significant.  The  conclusion  that  can  be  drawn  from  Tables  14 
through  19  is  that  the  higher  the  criterion  group,  the  more  excellent  and  good 
attributes  that  are  mentioned  by  the  evaluator.  The  lower  the  criterion  group, 
the  more  average,  poor,  and  poorest  incidents  of  behavior  that  are  cited  by  the 
evaluator. 

In  the  pilot  study  the  Mann-Whitney  U  test  was  used  to  test  the  null  hy¬ 
pothesis  that  there  were  no  differences  between  each  pair  of  criterion  groups 
on  each  of  the  67  quantitative  variables  derived  from  the  indexing  form.  The 
nonparametric  Mann-Whitney  U  test  was  used  in  preference  to  the  parametric  t 
test  of  difference  between  two  means  because  very  few  of  the  67  variables  were 
normally  distributed  and  the  t  test  assumes  normality  and  common  variance. 
However,  since  the  t  test  is  very  robust,  it  was  decided  subsequently  to  also 
compute  t  tests  of  mean  difference  for  the  pilot  study  sample  using  Program 
BMDP3D  in  the  library  of  Biomedical  Computer  Programs1^  at  the  UCLA  Health 
Sciences  Computing  Facility.  The  results  of  these  computer  runs  agreed  sub¬ 
stantially  with  the  results  obtained  using  the  Mann-Whitney  U  test,  primarily 
because  the  sample  size  of  the  various  criterion  groups  in  the  pilot  study  sam¬ 
ple  was  equal  and  large  (N=75).  The  probability  level  of  these  two  statisti¬ 
cal  tests  differed  occasionally  and  only  when  the  result  fell  near  the  bounda¬ 
ry  of  the  various  bench  marks  of  statistical  significance  (e.g.,  the  t  test 
might  be  significant  between  the  .01  and  .001  probability  levels  and  the  Mann- 
Whitney  U  test  might  be  significant  between  the  .05  and  .01  probability  lev¬ 
els)  . 


As  a  consequence  of  the  pilot  study  experience,  both  the  Mann-Whitney  U 
test  and  the  t  test  of  mean  difference  were  computed  on  each  of  the  67  varia¬ 
bles  for  the  cross  validation  and  generalization  samples.  These  computations 
were  made  for  each  pair  of  criterion  groups  in  both  the  evaluation  and  the  jus¬ 
tification  sections.  The  Mann-Whitney  U  tests  were  performed  at  the  computing 
facility  of  the  Naval  Electronics  Laboratory  Center  in  San  Diego  under  the  su¬ 
pervision  of  NPRDC.  The  t  tests  were  performed  at  the  UCLA  Health  Sciences 
Computing  Facility.  The  results  of  these  computer  runs  are  presented  in  Appen¬ 
dix  D.  This  appendix  reports  the  number  and  name  of  the  variable,  the  value 
of  U,  its  associated  absolute  z  value  (approximately  a  normal  deviate),  and 
the  corresponding  probability  level  for  a  2-tailed  test.  The  convention  that 
was  followed  in  the  Mann-Whitney  U  tests  was  to  enter  data  into  the  computer 
program  first  for  the  lower  criterion  group  in  the  comparison.  For  those  z 
values  that  achieved  statistical  significance,  the  direction  of  the  difference 
is  that  the  higher  criterion  group  evidenced  a  greater  frequency  or  greater 
weighted  frequency  of  the  variable.  The  Mann-Whitney  U  test  probability  level 
in  Appendix  D  is  followed  by  the  value  of  t  and  its  probability  level  for  a  2- 
tailed  test.  Thus  the  reader  can  compare  the  results  of  the  two  statistical 
tests  for  each  variable. 

Those  variables  resulting  in  a  statistically  significant  difference  in 
each  pairwise  criterion  group  comparison  have  been  extracted  from  Appendix  D 


r 
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and  summarized  in  four  tables- — Tables  20  through  23.  Only  those  variables 
showing  a  significant  difference  on  both  statistical  tests  have  been  extracted. 
Table  20  lists  the  statistically  significant  variables  for  the  evaluation  sec¬ 
tion  of  the  cross  validation  sample.  The  easiest  discrimination  to  be  made 
should  be  between  the  Lower  and  Upper  criterion  groups  since  they  are  the  most 
widely  separated  on  the  criterion  variable  19A-PERF0RMANCE  OF  DUTY.  Table  20 
reveals  that  eight  of  the  67  variables  showed  a  statistically  significant  dif¬ 
ference  between  the  Lower  and  Upper  criterion  groups.  Seven  of  these  eight 
variables  also  showed  a  statistically  significant  difference  between  the  Lower 
and  Upper  criterion  groups  in  the  pilot  study  sample,18  the  exception  being 
Total  Number  of  3  (New  1)  Weights  (Average).  In  evaluating  the  performance  of 
outstanding  chief  petty  officers  as  compared  to  average  CPOfs,  the  significant 
differences  reside  in  the  superiority  of  the  top  criterion  group  with  respect 
to  professionalism,  managerial  know-how,  more  awards  and  fewer  punishments, 
more  superlative  attributes,  and  fewer  qualified  statements  reflecting  fair  or 
poor  performance. 

By  comparison  Table  20  shows  that  there  were  eleven  statistically  signifi¬ 
cant  differences  between  the  Lower  and  Middle  criterion  groups  of  the  cross 
validation  sample  on  the  evaluation  section.  Only  one  of  these  eleven  varia¬ 
bles  showed  a  statistically  significant  difference  between  the  Lower  and  Mid¬ 
dle  criterion  groups  in  the  pilot  study  sample - Total  Number  of  2  (New  -1) 

Weights  (Poor).18  In  the  cross  validation  sample  the  areas  of  performance 
that  differentiated  average  CPOfs  from  those  who  are  marked  as  somewhat  supe¬ 
rior  in  performance  of  duty  were  professionalism,  cooperation,  responsiveness, 
resourcefulness,  and  productivity  and  achievement.  Ratees  in  the  middle  cri¬ 
terion  group  were  described  with  more  superlatives  and  with  fewer  qualified 
statements  of  fair  or  poor  performance. 

The  most  difficult  discrimination  to  be  made  is  that  between  the  Middle 
and  Upper  criterion  groups.  Table  20  shows  that  despite  this  difficulty, 
eight  statistically  significant  differences  resulted  from  the  comparison  on 
the  evaluation  section.  Only  two  of  these  eight  variables  showed  a  statisti¬ 
cally  significant  difference  between  the  Middle  and  Upper  criterion  groups  in 

the  pilot  study  sample - f  of  COOPERATION  and  wf  of  COOPERATION.18  In  the 

cross  validation  sample  the  characteristics  that  differentiated  outstanding 
CPO’s  from  their  slightly  less  qualified  colleagues  were  cooperation,  grooming 
and  attire,  resourcefulness,  and  productivity  and  achievement.  A  wider  range 
of  index  terms  was  used  to  index  the  Evaluation  Reports  of  outstanding  CPO’s 
which  also  resulted  in  more  3  (New  1)  weights  being  assigned  to  this  criterion 
group. 


Moving  now  to  a  consideration  of  the  justification  section  for  the  cross 
validation  sample  (Table  21),  all  but  five  of  the  67  variables  showed  a  statis¬ 
tically  significant  difference  between  the  Lower  and  Upper  criterion  groups 
and  all  but  eight  of  the  67  variables  showed  a  statistically  significant  dif¬ 
ference  between  the  Lower  and  Middle  criterion  groups.  This  plethora  of  sig¬ 
nificant  results  is  an  artifact  occasioned  by  the  requirement  to  write  an  in¬ 
dividual  justification  for  any  mark  in  the  top  10,  5,  or  1%  columns  of  Section 
19.  Despite  this  built-in  bias,  five  of  the  67  variables  were  not  statistical¬ 
ly  significant  for  the  Lower  versus  Upper  criterion  group  comparison:  f  of 
FLEXIBILITY,  wf  of  FLEXIBILITY,  wf  of  ORGANIZATION,  Total  Number  of  2  (New  -1) 
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STATISTICALLY  SIGNIFICANT  VARIABLES  RESULTING  FROM  PAIRWISE  CRITERION  GROUP  COMPARISONS 
ON  THE  JUSTIFICATION  SECTION  (19S)  FOR  THE  CROSS  VALIDATION  SAMPLE 


33 


u 

<y 

ex 

a 

£3 


H  m  fH  V  H 
e  g  w  4-i 

d  9  o  o  o 

>  wm  H  H  H 


a> 

d 

c 


M-l  £  M-< 


O 

OJ 

a 

d) 


wf  of  COOPERATION***  f  of  GROOMING  AND  ATTIRE***  Text** 

f  of  ENDURANCE***  wf  of  GROOMING  AND  ATTIRE**  Total  Number  of  Index  Terms 

wf  of  ENDURANCE**  Used** 


STATISTICALLY  SIGNIFICANT  VARIABLES  RESULTING  FROM  PAIRWISE  CRITERION  GROUP  COMPARISONS 
ON  THE  JUSTIFICATION  SECTION  (19S)  FOR  THE  CROSS  VALIDATION  SAMPLE 


34 


u 

0) 

a 

cx 

:=> 


w 

> 


QJ 

rH 

T3 

T3 


T3 

QJ 

D 

a 


c 

o 

o 


T3 


OM-J  O  44 
O  O  O  O 

44  44  44 

44  3  £44  £  4-4 


*  ^ 

55  O 

o  M 

>  M 

*  §*§ 

*  *  W  5 

*  *  W  O  M 

■K  W  U  H  N 

W  >  M  >  fc 

SM  >  Otf  W 

P ^  WH 
Otf  Q  W  CO  O 

Q  CO  p4 

4_l  44 

44  o  4-1  o  44 

O  O  O 

44  4-1 

44  £  44  £  44 


<u 

a 

Cl 

S3 


W 

> 

QJ 

* 

O 


44  O  44  O  44 

o  o  o 

44  44 

44  *£44  £  44 


44  CQ  QPQ44  044  Q44  O 

o  o  o  o 

44 

£  44  £  44  £  44  £ 


£  of  AWARDS  AND  PUNISHMENT*  wf  of  POTENTIAL*** 

wf  of  AWARDS  AND  PUNISHMENT** 


STATISTICALLY  SIGNIFICANT  VARIABLES  RESULTING  FROM  PAIRWISE  CRITERION  GROUP  COMPARISONS 
ON  THE  JUSTIFICATION  SECTION  (19S)  FOR  THE  CROSS  VALIDATION  SAMPLE 


35 


M 

o> 

a 

a 


■K 

*  * 
*  * 
*  On 
on  m 

CNI 


X 

CO 

X 

00 

a 

00 

3  ^ 

/*“N 

u 

■K 

0 

o 

CO 

CM 

rH 

3 

0) 

a> 

■K 

o 

M 

*H 

H 

rH 

> 

u 

X 

> 

> 

> 

73 

> 

< 

X 

4J 

0) 

<D 

<D 

CO 

X 

73 

X 

4-) 

X 

X 

X 

73 

Q) 

•h 

X 

rH 

"w ' 

N-/ 

w 

M 

73 

2 

w 

iH 

CO 

O 

3 

W 

X 

m 

CO 

X 

M 

• 

X 

H 

CO 

CO 

co 

H 

a> 

a> 

M-l 

M-l 

M-4 

M-l 

M-l 

> 

■K 

■K 

p 

iH 

rH 

O 

o 

O 

O 

O 

•K 

■K 

O 

H 

X 

X 

•K 

* 

* 

u 

■K 

W 

H 

cO 

cO 

u 

■K 

u 

■K 

u 

■K 

U 

U 

a> 

W 

H 

H 

*H 

•H 

0) 

*K 

a> 

■K 

Q) 

*K 

a> 

a) 

> 

H 

b 

H 

W 

U 

U 

X 

CO 

X 

CO 

X 

CO 

X 

•K 

X 

o 

X 

X 

Ph 

W 

oo 

cO 

CO 

s 

4J 

0 

u 

0 

4-i 

S  hc 

g 

& 

W 

00 

03 

> 

> 

p 

X 

y 

X 

3 

X 

3 

* 

3 

W 

52 

c/3 

<J 

X 

00  X 

00  X 

00  X 

4J 

X 

pc: 

< 

M-l 

M-l 

‘H 

X 

M-l 

O 

o 

rH 

0) 

rH 

0) 

rH 

a> 

rH 

0) 

rH 

m-i 

o 

M-l 

o 

CO 

X 

CO 

X 

CO  ^ 

CO 

H 

CO 

o 

o 

g 

■U 

4-> 

4-1 

4-1 

4-1 

4-4 

M-l 

3 

3 

0 

o 

o 

o 

o 

M-l 

3  M-l 

03 

03 

H 

H 

H 

H 

H 

* 

■K 

*K 

73 

a> 

co 

X 


■K 

•K  -K 
•K  * 
*  ON 

on  m 
CM 


HC 

X 

CO 

•K 

•K 

■K 

X 

oo 

a 

*K 

•K 

HC 

■K 

00 

3 

/■N 

/■% 

u 

* 

X 

* 

•K 

3 

o 

CO 

CM 

rH 

3 

0) 

X 

o 

* 

>< 

o 

u 

•H 

H 

M 

o 

H 

P* 

> 

u 

X 

> 

& 

3 

(1) 

M 

H 

> 

< 

X 

4-1 

(1) 

0) 

CO 

X 

a 

H 

<J 

< 

X 

4-» 

X 

X 

X 

73 

0) 

a 

< 

> 

X 

rH 

N-/ 

v-/ 

V-/ 

M 

73 

m 

X 

> 

H  -K 

w 

rH 

CO 

O 

3 

o 

M 

H  *  * 

w 

X 

m 

CO 

X 

H 

• 

• 

H 

O  *  * 

X 

H 

CO 

CO 

CO 

O 

*5=1 

* 

H 

a) 

a) 

mh 

M-l 

M-l 

M-l 

M-l 

V 

> 

* 

X 

* 

* 

O 

rH 

rH 

o 

O 

0 

O 

O 

* 

* 

W  3  H 

■K 

■K 

o 

H 

X 

X 

■K 

•K 

■K 

PH 

u 

* 

* 

w 

UHH 

■K 

W 

H 

CO 

CO 

u 

■K 

u 

* 

M 

■K 

M 

M 

0) 

* 

W 

o 

H  H  X 

W 

H 

H 

*H 

•H 

Q) 

■K 

a) 

■K 

0) 

■K 

0) 

0) 

V 

> 

w 

fc 

M 

>  X  W 

H 

X 

H 

W 

M 

X 

CO 

X 

CO 

X 

CO 

X 

■K 

X 

■K 

o 

> 

> 

Pi  W  H 

X 

PH 

W 

03 

CO 

CO 

a 

4-1 

0 

4-1 

a 

4-1 

a  * 

a 

* 

rH 

X 

M 

Pi 

Pi 

WHO 

w 

03 

03 

> 

> 

3 

X 

9 

X 

3 

X 

3 

■K 

3 

* 

o 

Pi 

X 

w 

03  O  PH 

□ 

52 

w 

<2 

X 

00  X 

00  X 

00  X 

4-1 

X 

73 

• 

Q 

03 

PH 

Pi 

< 

M-l 

M-l 

•H 

•rH 

X 

<U 

M-l 

M-l  M-l 

MM 

M-l 

O 

O 

rH 

0) 

rH 

0) 

rH 

0) 

rH 

Q> 

rH 

CO 

M-l 

O 

M-l 

O  M-l  O 

M-l 

O 

M-l 

o 

a 

a 

cO 

X 

cO 

X 

cO 

X 

CO 

H 

3 

X 

O 

O 

O 

O 

O 

U 

4J 

4J 

4-» 

4-1 

■K 

M-l 

M-l  M-l 

M-l 

M-l 

3 

3 

O 

O 

o 

O 

O 

M-l 

3  M-l 

3  M-l  3  M-l 

3  M-l 

> 

03 

03 

H 

H 

H 

H 

H 

**  .001  <  P  < 

***  P  <  .001 


36 


Weights  (Poor),  and  Total  Number  of  1  (New  -2)  Weights  (Poorest),  No  2  or  1 
weights  were  used  in  indexing  the  Upper  criterion  group  and  only  two  2  weights 
and  one  1  weight  were  used  in  indexing  the  Lower  criterion  group  (see  Table 
15).  Eight  of  the  67  variables  were  not  statistically  significant  for  the  Low¬ 
er  versus  Middle  criterion  group  comparison:  f  of  ENDURANCE,  wf  of  ENDURANCE, 
f  of  FLEXIBILITY,  wf  of  FLEXIBILITY,  f  of  AWARDS  AND  PUNISHMENT,  wf  of  INTEL¬ 
LECTUAL  FUNCTIONING,  Total  Number  of  2  (New  -1)  Weights  (Poor),  and  Total  Num¬ 
ber  of  1  (New  -2)  Weights  (Poorest).  Only  two  2  weights  and  no  1  weights  were 
used  in  indexing  the  Middle  criterion  group  compared  to  two  2  weights  and  one 
1  weight  for  the  Lower  criterion  group  (see  Table  15) . 

The  only  really  cogent  comparison  for  the  justification  section  is  be¬ 
tween  the  Upper  and  the  Middle  criterion  groups  because  both  of  these  groups 
required  justification  comments.  Table  21  shows  that  21  of  the  67  variables 
evidenced  a  statistically  significant  difference  between  these  two  criterion 
groups,  eight  of  which  also  showed  a  statistically  significant  difference  be¬ 
tween  the  Upper  and  Middle  criterion  groups  in  the  pilot  study  sample - wf  of 

SKILLS  AND  ABILITIES,  f  of  PRODUCTIVITY  AND  ACHIEVEMENT,  wf  of  PRODUCTIVITY 
AND  ACHIEVEMENT,  Sum  of  Variables  1  through  29,  Sum  of  Variables  31  through  59, 
Total  Number  of  5  (New  3)  Weights,  Total  Number  of  Words  in  Text,  and  Total 
Number  of  Index  Terms  Used.19  When  the  evaluator  is  required  to  justify  his 
marks  in  evaluating  a  ratee,  he  apparently  calls  out  certain  areas  of  perform¬ 
ance  that  distinguish  the  ratee  in  the  Upper  criterion  group  from  his  slightly 
less  qualified  colleague  in  the  Middle  criterion  group.  In  the  cross  valida¬ 
tion  sample  skills  and  abilities  as  well  as  productivity  and  achievement  were 
the  differentiating  areas  of  performance.  The  ratee  in  the  Upper  criterion 
group  also  had  bestowed  upon  him  more  superlative  adjectives  and  adverbs.  All 
three  criterion  group  comparisons  on  the  justification  section  of  the  Evalua¬ 
tion  Report  showed  significant  differences  on  both  Total  Number  of  Words  in 
Text  and  Total  Number  of  Index  Terms  Used. 

Table  22  lists  the  statistically  significant  variables  for  the  evaluation 
section  of  the  generalization  sample.  Again,  the  easiest  discrimination  to  be 
made  should  be  between  the  Lower  and  Upper  criterion  groups  since  they  are  the 
most  widely  separated  on  the  criterion  variable  19A-PERF0RMANCE  OF  DUTY.  Only 
four  of  the  67  variables  showed  a  statistically  significant  difference  between 

the  Lower  and  Upper  criterion  groups - f  of  AWARDS  AND  PUNISHMENT,  wf  of  AWARDS 

AND  PUNISHMENT,  Total  Number  of  5  (New  3)  Weights  (Excellent),  and  Total  Number 
of  2  (New  -1)  Weights  (Poor).  All  four  of  these  variables  also  showed  a  sta¬ 
tistically  significant  difference  between  the  Lower  and  Upper  criterion  groups 
on  the  evaluation  section  of  the  cross  validation  sample. 

By  comparison  Table  22  shows  that  there  were  six  statistically  significant 
differences  between  the  Lower  and  Middle  criterion  groups  of  the  generalization 
sample  on  the  evaluation  section.  Only  one  of  these  six  variables  also  showed 
a  statistically  significant  difference  between  the  Lower  and  Middle  criterion 

groups  on  the  evaluation  section  of  the  cross  validation  sample - Total  Number 

of  2  (New  -1)  Weights  (Poor).  Four  statistically  significant  differences  re¬ 
sulted  from  the  comparison  between  the  Middle  and  Upper  criterion  groups  on 
the  evaluation  section  of  the  generalization  sample - f  of  POTENTIAL,  wf  of  PO¬ 

TENTIAL,  Total  Number  of  5  (New  3)  Weights  (Excellent),  and  Total  Number  of  2 
(New  -1)  Weights  (Poor).  None  of  these  four  variables  overlapped  with  those 
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that  were  statistically  significant  for  the  Middle  versus  Upper  criterion 
group  comparison  on  the  evaluation  section  of  the  cross  validation  sample. 

These  results  may  indicate  that  the  findings  in  the  cross  validation  sample 
may  not  map  particularly  well  onto  the  findings  in  the  generalization  sample, 
probably  because  of  the  differences  in  the  occupational  ratings  represented  in 
these  two  samples.  This  issue  will  be  dealt  with  more  thoroughly  later  in  this 
section  when  the  results  of  the  stepwise  discriminant  analyses  are  discussed. 

Moving  now  to  a  consideration  of  the  justification  section  for  the  gener¬ 
alization  sample  (Table  23),  all  but  nine  of  the  67  variables  showed  a  statis¬ 
tically  significant  difference  between  the  Lower  and  Upper  criterion  groups. 
Four  of  these  nine  variables  also  showed  a  lack  of  statistical  significance 
between  the  Lower  and  Upper  criterion  groups  on  the  justification  section  of 
the  cross  validation  sample - f  of  FLEXIBILITY,  wf  of  FLEXIBILITY,  Total  Num¬ 

ber  of  2  (New  -1)  Weights  (Poor) ,  and  Total  Number  of  1  (New  -2)  Weights 
(Poorest).  All  but  15  of  the  67  variables  showed  a  statistically  significant 
difference  between  the  Lower  and  Middle  criterion  groups  on  the  justification 
section  of  the  generalization  sample  (Table  23).  Five  of  these  15  variables 
also  showed  a  lack  of  statistical  significance  between  the  Lower  and  Middle 

criterion  groups  on  the  justification  section  of  the  cross  validation  sample - 

f  of  FLEXIBILITY,  wf  of  FLEXIBILITY,  f  of  AWARDS  AND  PUNISHMENT,  Total  Number 
of  2  (New  -1)  Weights  (Poor),  and  Total  Number  of  1  (New  -2)  Weights  (Poorest). 

As  pointed  out  earlier,  the  only  really  cogent  comparison  for  the  justifi¬ 
cation  section  is  between  the  Upper  and  the  Middle  criterion  groups  because 
both  of  these  groups  required  justification  comments.  Table  23  shows  that  38 
of  the  67  variables  evidenced  a  statistically  significant  difference  between 
these  two  criterion  groups  in  the  generalization  sample,  17  of  which  also 
showed  a  statistically  significant  difference  between  the  Upper  and  Middle  cri¬ 
terion  groups  in  the  cross  validation  sample.  This  finding  suggests  that  there 
is  partial  overlap  between  these  two  samples.  The  nonoverlapping  areas  may  be 
attributed  to  a  difference  in  the  nature  of  the  occupational  ratings  represent¬ 
ed  in  the  cross  validation  and  the  generalization  samples  and/or  to  unrelia¬ 
bility  in  the  indexing  procedure,  although  the  results  of  the  reliability  study 
presented  in  Section  5  suggest  that  differences  among  the  four  occupational 
ratings  provide  a  more  reasonable  explanation  for  these  results  than  unrelia¬ 
bility.  Both  of  these  issues  will  be  discussed  more  fully  later  in  this  re¬ 
port. 


In  every  pairwise  criterion  group  comparison  shown  in  Tables  20  through 
23,  one  or  more  of  the  five  variables  involving  Total  Number  of  5  (New  3),  4 
(New  2),  3  (New  1),  2  (New  -1),  or  1  (New  -2)  Weights  proved  to  be  signifi¬ 
cantly  differentiating.  This  finding  supports  the  results  of  the  correla¬ 
tional  analysis  presented  in  Appendix  C  in  which  the  system  used  for  weighting 
Variables  31  through  59  was  highlighted  as  an  important  aspect  of  the  content 
analysis  methodology. 

Thus  far  in  this  report  the  67  quantitative  variables  derived  from  the 
indexing  form  have  been  considered  individually  as  potential  discriminators 
among  the  three  criterion  groups  in  the  evaluation  and  justification  sections 
of  the  various  samples.  However,  it  is  of  considerable  interest  to  learn  how 
well  this  set  of  variables,  used  in  optimal  combination,  can  classify  each  of 
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the  research  samples  into  correct  criterion  group.  Therefore,  the  three  sam¬ 
ples  were  also  analyzed  by  Program  BMD07M  in  the  library  of  Biomedical  Comput¬ 
er  Programs13  at  the  UCLA  Health  Sciences  Computing  Facility.  This  program 
performs  a  multiple  discriminant  analysis  in  a  stepwise  manner.  At  each  step 
one  variable  is  entered  into  the  set  of  discriminating  variables  or  a  variable 
is  deleted  if  its  F  value  becomes  too  low.  At  the  option  of  the  user,  a  clas¬ 
sification  matrix  is  computed  and  printed  after  those  steps  specified  by  the 
user.  This  option  permits  the  user  to  determine  if  the  classification  process 
tends  to  converge  to  perfect  classification  or  to  maximize  at  some  midway  step 
and  then  disintegrate  as  more  variables  are  added  to  the  discriminant  function. 

A  special  feature  of  Program  BMD07M  allows  new  cases  to  be  classified  by 
the  discriminant  functions  generated  on  the  original  sample.  This  feature  was 
used  to  perform  two  cross  validation  studies  of  the  AT’s  and  BTfs  combined  and 
also  of  the  ATfs  and  BTTs  considered  separately.  In  the  first  study,  the  cross 
validation  sample  and  the  pilot  study  sample  were  both  classified  using  the 
cross  validation  sample  discriminant  functions.  In  the  second  study,  the  pilot 
study  sample  and  the  cross  validation  sample  were  both  classified  using  the  pi¬ 
lot  study  sample  discriminant  functions.  Figure  5  portrays  the  accuracy  of 
classifying  the  cross  validation  sample  and  the  pilot  study  sample  into  correct 
criterion  group  using  the  cross  validation  sample  discriminant  functions  for 
the  evaluation  section  (19R) .  For  the  first  three  steps  of  the  discriminant 
analysis  the  curves  for  the  two  samples  are  very  close.  Then,  as  the  discrimi¬ 
nant  analysis  progresses  step  by  step,  the  two  curves  begin  to  separate  with 
the  accuracy  of  classifying  the  pilot  study  sample  being  considerably  less  than 
the  accuracy  of  classifying  the  cross  validation  sample.  It  was  expected,  of 
course,  that  classification  of  the  pilot  study  sample  using  the  cross  valida¬ 
tion  sample  discriminant  functions  would  be  less  accurate  than  classification 
of  the  cross  validation  sample  itself.  Nonetheless,  the  pilot  study  sample 
curve  tracks  the  behavior  of  the  cross  validation  sample  curve  quite  faithfully 
and  even  drops  off  precipitously  at  Step  64  as  classification  of  the  cross 
validation  sample  disintegrates.  Figure  6  portrays  the  accuracy  of  classifying 
the  cross  validation  sample  and  the  pilot  study  sample  into  correct  criterion 
group  using  the  cross  validation  sample  discriminant  functions  for  the  justi¬ 
fication  section  (19S) .  As  in  Figure  5  the  curves  for  the  two  samples  remain 
very  close  for  the  first  five  steps  after  which  they  begin  to  diverge.  Again, 
less  accuracy  was  achieved  in  classifying  the  pilot  study  sample  using  the 
cross  validation  sample  discriminant  functions  than  in  classifying  the  cross 
validation  sample  itself.  However,  it  should  be  noted  that  for  both  samples 
better  classification  accuracy  was  achieved  on  the  justification  section  than 
on  the  evaluation  section.  This  is  more  clearly  shown  in  Table  24. 

Table  24  displays  the  best  classification  that  was  achieved  on  the  evalua¬ 
tion  section  and  the  justification  section  of  the  cross  validation  sample  and 
the  pilot  study  sample  using  the  cross  validation  sample  discriminant  func¬ 
tions.  The  underlined  diagonal  elements  of  the  four  classification  matrices 
portrayed  in  Table  24  represent  agreement  between  the  statistical  classifica¬ 
tion  into  criterion  group  and  actual  criterion  group  membership;  the  off- 
diagonal  elements  represent  disagreement.  The  total  number  of  statistical 
classifications  matching  actual  classification  is  obtained  by  summing  the  di¬ 
agonal  elements  of  each  matrix,  shown  in  Table  24  as  the  underlined  diagonal 
sum.  The  step  in  the  discriminant  analysis  at  which  this  best  classification 
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Number  of  Cases  Classified  Correctly 


Figure  6.  Accuracy  of  Classifying  the  Cross  Validation  Sample  (N«222)  and  the  Pilot 
Study  Sample  (N=224)  into  Correct  Criterion  Group  Using  the  Cross  Valida¬ 
tion  Sample  Discriminant  Functions  (Justification  Section  -  19S) 
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TABLE  24 

BEST  CLASSIFICATION  INTO  THE  THREE  CRITERION  GROUPS 
USING  THE  CROSS  VALIDATION  SAMPLE  DISCRIMINANT  FUNCTIONS 


EVALUATION  SECTION  -  19R 
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was  achieved  is  also  shown  in  Table  24  and  corresponds  to  the  maximum  point  on 
the  curves  shown  in  Figures  5  and  6.  Of  the  222  cases  in  the  cross  validation 
sample,  189  (85%)  were  classified  correctly  at  Step  40  in  the  discriminant 
analysis  of  the  justification  section  of  the  cross  validation  sample.  Of  the 
224  cases  in  the  pilot  study  sample,  158  (71%)  were  classified  correctly  on 
the  justification  section  with  the  cross  validation  sample  discriminant  func¬ 
tions,  this  maximum  classification  accuracy  occurring  at  the  fourth  step.  It 
should  be  pointed  out  that  this  presentation  of  the  discriminant  analysis  re¬ 
sults  assumes  that  the  criterion  of  actual  group  membership  is  perfect  where 
in  fact  the  possibility  does  exist  that  some  of  the  members  of  the  sample  were 
given  inflated  marks  on  19A-PERF0RMANCE  OF  DUTY,  and  consequently,  were  as¬ 
signed  to  an  incorrect  criterion  group.  Also,  it  is  possible  that  narrative 
comments  and  evaluation  marks  may  tap  different  aspects  of  performance. 

Figures  7  and  8  portray  the  accuracy  of  classifying  the  pilot  study  sam¬ 
ple  and  the  cross  validation  sample  into  correct  criterion  group  using  the  pi¬ 
lot  study  sample  discriminant  functions.  Figure  7  shows  these  results  for  the 
evaluation  section  (19R)  and  Figure  8  shows  them  for  the  justification  section 
(19S) .  As  in  the  reverse  cross  validation  shown  in  Figures  5  and  6,  the  two 
curves  remain  close  together  for  the  first  five  steps  or  so  and  then  they  di¬ 
verge  with  the  pilot  study  sample  being  classified  more  accurately  on  its  own 
discriminant  functions  than  the  cross  validation  sample.  Once  again  better 
classification  accuracy  was  achieved  for  both  samples  on  the  justification 
section.  This  is  shown  more  clearly  in  Table  25.  Of  the  224  cases  in  the  pi¬ 
lot  study  sample,  200  (89%)  were  classified  correctly  at  Step  42  in  the  dis¬ 
criminant  analysis  of  the  justification  section  of  the  pilot  study  sample.  Of 
the  222  cases  in  the  cross  validation  sample,  149  (67%)  were  classified  cor¬ 
rectly  on  the  justification  section  with  the  pilot  study  sample  discriminant 
functions,  this  maximum  classification  accuracy  occurring  at  the  third  step. 

It  was  hypothesized  that  better  classification  would  be  achieved  in  using 
Program  BMD07M  if  the  two  occupational  ratings  represented  in  each  of  the 
three  research  samples  were  analyzed  separately.  The  skills  needed  to  achieve 
superior  performance  may  be  quite  different  for  Aviation  Electronics  Techni¬ 
cians  than  for  Boilermen.  Therefore,  two  cross  validation  studies  also  were 
conducted  for  the  AT’s  and  for  the  BTfs.  Figures  9  and  10  portray  the  accura¬ 
cy  of  classifying  the  cross  validation  AT’s  and  the  pilot  study  AT’s  into  cor¬ 
rect  criterion  group  using  the  cross  validation  ATfs  discriminant  functions. 
Figure  9  shows  these  results  for  the  evaluation  section  (19R)  and  Figure  10 
shows  them  for  the  justification  section  (19S).  In  both  of  these  figures  the 
two  curves  remain  close  together  in  the  earlier  steps  and  then  diverge  with 
the  cross  validation  AT’s  being  classified  more  accurately  on  their  own  dis¬ 
criminant  functions  than  the  pilot  study  ATfs.  As  in  the  total  cross  valida¬ 
tion  and  pilot  study  samples,  better  classification  accuracy  was  achieved  for 
both  samples  on  the  justification  section.  Table  26  shows  this  more  clearly. 
Of  the  138  cross  validation  AT’s,  129  (93%)  were  classified  correctly  at  Step 
46  in  the  discriminant  analysis  of  the  justification  section  for  the  cross 
validation  AT’s.  Of  the  144  pilot  study  AT’s,  96  (67%)  were  classified  cor¬ 
rectly  on  the  justification  section  with  the  cross  validation  AT’s  discrimi¬ 
nant  functions,  this  maximum  classification  accuracy  occurring  on  the  first 
step. 
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Figure  8.  Accuracy  of  Classifying  the  Pilot  Study  Sample  (N*224)  and  the  Cross 

Validation  Study  Sample  (N=222)  into  Correct  Criterion  Group  Using  the 
Pilot  Study  Sample  Discriminant  Functions  (Justification  Section  -  19S) 
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TABLE  25 

BEST  CLASSIFICATION  INTO  THE  THREE  CRITERION  GROUPS 
USING  THE  PILOT  STUDY  SAMPLE  DISCRIMINANT  FUNCTIONS 


EVALUATION  SECTION  -  19R 


PILOT  STUDY  SAMPLE  (N-224) 


CROSS  VALIDATION  SAMPLE  (N-222) 
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JUSTIFICATION  SECTION  -  19S 
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Figure  9.  Accuracy  of  Classifying  the  Cross  Validation  ATfs  (N-138)  and  the  Pilot 
Study  ATfs  (N«144)  into  Correct  Criterion  Group  Using  the  Cross  Valida¬ 
tion  ATfs  Discriminant  Functions  (Evaluation  Section  -  19R) 
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TABLE  26 

BEST  CLASSIFICATION  INTO  THE  THREE  CRITERION  GROUPS 
USING  THE  CROSS  VALIDATION  AT's  DISCRIMINANT  FUNCTIONS 


EVALUATION  SECTION  -  19R 
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The  results  of  the  reverse  cross  validation  for  the  pilot  study  AT’s  and 
the  cross  validation  AT’s  are  shown  in  Figures  11  and  12,  Figure  11  portrays 
the  accuracy  of  classifying  the  pilot  study  ATfs  and  the  cross  validation  ATf s 
into  correct  criterion  group  on  the  evaluation  section  using  the  pilot  study 
ATfs  discriminant  functions.  Figure  12  depicts  parallel  results  for  the  jus¬ 
tification  section.  It  now  seems  abundantly  clear  that  the  two  curves  remain 
close  together  over  the  first  five  steps  or  so  and  then  diverge.  The  curve 
for  the  sample  classified  by  its  own  discriminant  functions  continues  to  rise 
to  some  maximum  point  of  classification  accuracy,  usually  between  Steps  40  and 
50.  The  best  classification  accuracy  for  the  sample  being  classified  by  the 
other  sample’s  discriminant  functions  is  achieved  very  early,  typically  before 
Step  5  after  which  the  classification  accuracy  begins  to  deteriorate  and  the 
two  curves  diverge.  Comparing  Figure  11  with  Figure  12  it  can  be  seen  again 
that  better  classification  accuracy  was  achieved  for  both  samples  on  the  jus¬ 
tification  section.  Table  27  presents  the  classification  matrices  illustrat¬ 
ing  this  outcome  more  clearly.  Of  the  144  pilot  study  AT’s,  137  (95%)  were 
classified  correctly  at  Step  58  in  the  discriminant  analysis  of  the  justifica¬ 
tion  section  for  the  pilot  study  AT’s.  Of  the  138  cross  validation  AT’s,  91 
(66%)  were  classified  correctly  on  the  justification  section  with  the  pilot 
study  AT’s  discriminant  functions,  this  maximum  classification  accuracy  occur¬ 
ring  on  the  third  step. 

Two  cross  validation  studies  also  were  conducted  for  the  cross  validation 
BT’s  and  the  pilot  study  BT’s,  and  the  results  from  these  studies  continue  to 
support  the  picture  that  has  already  emerged.  Figure  13  portrays  the  accuracy 
of  classifying  the  cross  validation  BT’s  and  the  pilot  study  BT’s  into  correct 
criterion  group  on  the  evaluation  section  using  the  cross  validation  BT’s  dis¬ 
criminant  functions.  Parallel  results  for  the  justification  section  are  shown 
in  Figure  14.  Perfect  classification  accuracy  was  achieved  on  the  justifica¬ 
tion  section  at  Step  46  for  the  84  cross  validation  BT’s  using  their  own  dis¬ 
criminant  functions  (see  Table  28).  Of  the  80  pilot  study  BT’s,  54  (68%)  were 
classified  correctly  on  the  justification  section  with  the  cross  validation 
BT’s  discriminant  functions,  this  maximum  classification  accuracy  occurring  at 
Step  8. 

In  the  reverse  cross  validation  for  the  pilot  study  BT’s  and  the  cross 
validation  BT’s  classified  by  the  pilot  study  BT’s  discriminant  functions,  the 
results  are  similar.  Figure  15  portrays  the  curve  of  classification  accuracy 
for  the  evaluation  section  and  Figure  16  depicts  the  classification  accuracy 
curve  for  the  justification  section.  Table  29  presents  the  four  classifica¬ 
tion  matrices  corresponding  to  Figures  15  and  16  in  which  the  best  classifica¬ 
tion  achieved  is  portrayed.  All  80  pilot  study  BT’s  were  classified  correctly 
on  the  justification  section  by  their  own  discriminant  functions.  Of  the  84 
cross  validation  BT’s,  55  (65%)  were  classified  correctly  on  the  justification 
section  with  the  pilot  study  BT’s  discriminant  functions,  this  maximum  clas¬ 
sification  accuracy  occurring  at  the  first  step. 

The  results  of  the  two  cross  validation  studies  were  very  similar  for  the 
total  cross  validation  sample  and  the  total  pilot  study  sample,  for  the  cross 
validation  AT’s  and  the  pilot  study  AT’s,  and  for  the  cross  validation  BT’s 
and  the  pilot  study  BT’s.  These  findings  support  the  expectation  held  at  the 
outset  of  this  investigation  that  it  would  be  possible  to  index  the  cross 
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Figure  14.  Accuracy  of  Classifying  the  Cross  Validation  BT's  (N=84)  and  the  Pilot 
Study  BT's  (N=80)  into  Correct  Criterion  Group  Using  the  Cross  Valida¬ 
tion  BT's  Discriminant  Functions  (Justification  Section  -  19S) 
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Figure  16.  Accuracy  of  Classifying  the  Pilot  Study  BT's  (N»80)  and  the  Cross 

Validation  BT's  (N=84)  into  Correct  Criterion  Group  Using  the  Pilot 
Study  BT's  Discriminant  Functions  (Justification  Section  -  19S) 
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validation  sample  in  the  blind,  without  knowledge  of  criterion  group  member¬ 
ship,  and  achieve  as  good  classification  accuracy  as  was  achieved  with  the  pi¬ 
lot  study  sample  where  criterion  group  membership  was  known  to  the  indexer. 
Further,  it  can  be  concluded  that  better  classification  into  the  three  crite¬ 
rion  groups  using  an  optimum  combination  of  the  67  quantitative  variables  is 
achieved  when  the  two  occupational  ratings  represented  in  the  pilot  study  sam¬ 
ple  and  the  cross  validation  sample  are  treated  separately.  The  superior  clas¬ 
sification  accuracy  achieved  for  the  BT's  compared  to  the  AT's  Indicates  that 
the  Aviation  Electronics  Technician  rating  may  represent  a  more  varied  amal¬ 
gamation  of  technical  activities  than  the  Boilerman  rating.  These  findings 
suggest  that  classification  procedures  based  on  the  content  analysis  methodolo¬ 
gy  developed  in  this  research  should  be  tailored  to  specific  occupations.  In 
all  of  the  results  presented  thus  far,  better  classification  was  achieved  in 
the  discriminant  analyses  of  the  justification  section  compared  to  the  evalua¬ 
tion  section.  In  the  various  cross  validation  runs  reported,  it  appears  that 
the  best  classification  accuracy  that  can  be  achieved  on  a  second  sample  using 
discriminant  functions  developed  on  the  first  sample  is  65  to  70  percent. 

This  level  of  accuracy  is  achieved  early  in  the  stepwise  discriminant  analysis 
procedure,  typically  by  the  fifth  step.  This  finding  suggests  that  the  varia¬ 
bles  selected  by  the  discriminant  analysis  program  at  the  first  five  steps  are 
crucial  variables.  Later  in  this  section  the  variables  selected  for  the  first 
15  steps  in  each  of  the  cross  validation  runs  will  be  enumerated.  But  before 
turning  to  this  enumeration,  the  results  in  classifying  the  generalization  sam¬ 
ple  (CS's  and  RM's)  will  be  discussed. 

Discriminant  analyses  also  were  performed  for  the  CS’s  and  RM's  combined 
(the  total  generalization  sample)  as  well  as  for  each  of  these  two  occupation¬ 
al  ratings  separately.  The  accuracy  of  classifying  the  generalization  sample 
into  correct  criterion  group  is  shown  in  Figures  17  and  18,  first  for  the 
evaluation  section  and  then  for  the  justification  section.  Better  classifica¬ 
tion  accuracy  was  achieved  for  the  justification  section  (see  Table  30)  than 
for  the  evaluation  section  where  190  of  the  222  cases  (86%)  in  the  justifica¬ 
tion  portion  of  the  generalization  sample  were  classified  correctly. 

When  the  CS's  and  RM's  were  considered  separately,  better  classification 
accuracy  was  achieved.  Referring  to  Table  30,  58  of  the  60  generalization 
CS's  (97%)  were  classified  correctly  on  the  evaluation  section.  All  60  gener¬ 
alization  CS's  were  classified  correctly  on  the  justification  section.  The 
curves  depicting  the  accuracy  of  classifying  the  CS's  on  both  the  evaluation 
section  and  the  justification  section  are  shown  in  Figures  19  and  20.  The  162 
RM's  in  the  generalization  sample  were  not  classified  as  accurately  as  the  CS's, 
although  better  classification  accuracy  for  the  RM's  was  achieved  on  the  justi¬ 
fication  section  where  144  cases  were  classified  correctly  (89%) .  These  re¬ 
sults  are  shown  in  Table  30.  The  curves  portraying  the  accuracy  of  classifying 
the  RM's  on  both  the  evaluation  section  and  the  justification  section  are  shown 
in  Figures  21  and  22.  Again  the  superiority  of  the  justification  section  in 
providing  discriminating  narrative  comments  is  demonstrated.  On  the  justifica¬ 
tion  section  all  60  CS's  were  classified  correctly  by  their  discriminant  func¬ 
tions  and  89  percent  of  the  RM's  were  classified  correctly  by  their  discrimi¬ 
nant  functions.  Eighty-six  percent  of  the  total  generalization  sample  was 
classified  correctly  on  the  justification  section  by  the  discriminant  functions 
based  on  CS's  and  RM's  combined.  This  suggests  that  the  occupational  rating, 
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Figure  18.  Accuracy  of  Classifying  the  Generalization  Sample  (N-222) 
into  Correct  Criterion  Group  (Justification  Section  -  19S) 


64 


TABLE  30 

BEST  CLASSIFICATION  INTO  THE  THREE  CRITERION  GROUPS 
FOR  THE  GENERALIZATION  SAMPLE 


EVALUATION  SECTION  -  19R 

CS's  &  RM's  COMBINED  (N=222) 
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Figure  20.  Accuracy  of  Classifying  the  Generalization  CS's  (N*60) 

into  Correct  Criterion  Group  (Justification  Section  -  19S) 
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Radioman,  may  be  more  heterogeneous  a,nd  require  a  greater  variety  of  skills 
than  the  Commissaryman  rating  just  as  the  Aviation  Electronics  Technician  rat¬ 
ing  may  represent  a  more  varied  amalgamation  of  technical  activities  than  the 
Boilerman  rating.  The  case  for  treating  each  occupational  rating  separately 
seems  to  be  supported  additionally  by  the  classification  results  for  the  gen¬ 
eralization  sample.  Moreover,  it  is  evident  that  the  content  analysis  method¬ 
ology  developed  initially  on  the  pilot  study  sample  consisting  of  AT’s  and 
BT’s  was  generalizable  to  a  new  sample  consisting  of  two  different  occupation¬ 
al  ratings,  viz.,  CS’s  and  RM’s. 

A  most  interesting  set  of  results  is  revealed  by  an  examination  of  which 
variables  were  selected  by  the  stepwise  discriminant  analysis  program  for  the 
first  15  steps  in  each  of  the  computer  runs  that  were  made.  Tables  31  and  32 
show  the  results  for  the  total  cross  validation  sample  (AT’s  and  BT’s  combined) 
and  the  total  pilot  study  sample  (AT’s  and  BT’s  combined).  The  results  are 
presented  separately  for  the  evaluation  section  and  the  justification  section. 
If  the  variable  was  one  of  the  first  15  variables  selected  in  both  of  the  sam¬ 
ples  for  each  section  considered  separately,  it  has  been  flagged  with  an  as¬ 
terisk  in  Tables  31  and  32.  Therefore,  it  can  be  determined  quickly  that  six 
of  the  first  15  variables  selected  by  the  stepwise  discriminant  analysis  pro¬ 
gram  in  the  cross  validation  sample  and  the  pilot  study  sample  for  the  evalua¬ 
tion  section  were  the  same,  namely.  Total  Number  of  5  (New  3)  Weights,  Total 
Number  of  2  (New  -1)  Weights,  f  of  COOPERATION,  wf  of  AWARDS  AND  PUNISHMENT, 
f  of  LEADERSHIP  AND  DIRECTING,  and  wf  of  TECHNICAL  SKILLS.  Furthermore,  Total 
Number  of  5  (New  3)  Weights  was  the  first  variable  selected  in  both  samples 
for  the  evaluation  section,  with  Total  Number  of  2  (New  -1)  Weights  being  the 
second  variable  selected  in  the  pilot  study  sample  and  the  third  variable  se¬ 
lected  in  the  cross  validation  sample.  It  appears  that  the  incidence  of  super¬ 
lative  adjectives  and  adverbs  describing  excellent  performance  (5  Weights)  and 
the  incidence  of  negatively  comparative  adjectives  and  adverbs  describing  in¬ 
adequate  performance  (2  Weights)  constitute  key  discriminating  variables  in 
analyzing  the  content  of  narrative  comments  written  for  the  evaluation  section 
in  these  two  samples. 

Of  more  interest  is  the  justification  section  since  without  exception 
better  classification  was  achieved  in  the  content  analysis  of  the  narrative 
comments  written  for  the  justification  section.  Four  of  the  first  15  varia¬ 
bles  selected  by  the  stepwise  discriminant  analysis  program  for  the  justifica¬ 
tion  section  in  the  total  cross  validation  sample  (ATfs  and  BTfs  combined)  and 
the  total  pilot  study  sample  (AT’s  and  BT’s  combined)  were  the  same.  Total 
Number  of  Index  Terms  Used  was  the  first  variable  selected  in  both  samples. 

Sum  of  Variables  1  through  29  was  the  second  variable  selected  in  the  cross 
validation  sample  and  the  fourth  variable  selected  in  the  pilot  study  sample, 
wf  of  PRODUCTIVITY  AND  ACHIEVEMENT  was  the  third  variable  selected  in  the  pi¬ 
lot  study  sample  and  the  fourth  variable  selected  in  the  cross  validation  sam¬ 
ple.  f  of  DRIVE  was  the  fifth  variable  selected  in  the  pilot  study  sample  and 
the  seventh  variable  selected  in  the  cross  validation  sample. 

Similar  tables  of  results  also  were  prepared  for  the  AT’s  and  BT’s  con¬ 
sidered  separately  in  the  cross  validation  and  pilot  study  samples.  Tables  33 
and  34  show  the  first  15  variables  selected  by  the  stepwise  discriminant  analy¬ 
sis  program  for  the  cross  validation  AT’s  and  the  pilot  study  AT’s  in  the 
evaluation  and  justification  sections.  As  in  Tables  31  and  32,  an  asterisk 
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TABLE  31 


VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  CROSS  VALIDATION  SAMPLE 


Step  Evalu ation  Section 

1  Total  Number  of  5  (New  3) 

Weights* 

2  Total  Number  of  Index  Terms 

Used 

3  Total  Number  of  2  (New  -1) 

Weights* 

4  f  of  COOPERATION* 


5 


6 

7 

8 
9 


10 

11 

12 


f  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 

wf  of  RESOURCEFULNESS 

f  of  PROFESSIONALISM 

wf  of  AWARDS  AND  PUNISHMENT* 

Total  Number  of  4  (New  2) 
Weights 

f  of  LEADERSHIP  AND  DIRECTING* 
wf  of  RESPONSIVENESS 
f  of  TECHNICAL  SKILLS 


13  f  of  POTENTIAL 

14  wf  of  TECHNICAL  SKILLS* 

15  f  of  RESPONSIVENESS 


Justification  Section 

Total  Number  of  Index  Terms 
Used* 

Sum  of  Variables  1  through  29* 

wf  of  ORGANIZATION 

wf  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT* 

f  of  ENDURANCE 

wf  of  TECHNICAL  SKILLS 
f  of  DRIVE* 

f  of  INTELLECTUAL  FUNCTIONING 
f  of  ASSET  TO  THE  NAVY 

wf  of  POTENTIAL 

f  of  TECHNICAL  SKILLS 

f  of  CONDUCT,  INTEGRITY,  AND 
PRIDE 

wf  of  AWARDS  AND  PUNISHMENT 
f  of  LEADERSHIP  AND  DIRECTING 
wf  of  LEADERSHIP  AND  DIRECTING 


Also  selected  by  the  stepwise  discriminant  analysis  program  for  this  section 
in  the  pilot  study  sample. 
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TABLE  32 

VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  PILOT  STUDY  SAMPLE 


Step  Evaluation  Section 

1  Total  Number  of  5  (New  3) 

Weights* 

2  Total  Number  of  2  (New  -1) 

Weights* 

3  f  of  LEADERSHIP  AND  DIRECTING* 

4  wf  of  TECHNICAL  SKILLS* 

5  Total  Number  of  1  (New  -2) 

Weights 

6  wf  of  AWARDS  AND  PUNISHMENT* 

7  wf  of  MANAGEMENT  FUNCTIONS 

8  f  of  SERVICE  MOTIVATION 

9  f  of  COOPERATION* 

10  Total  Number  of  3  (New  1) 

Weights 

11  wf  of  REPRESENTATION 

12  wf  of  CONTROLLING 

13  wf  of  COMMUNICATION 

14  wf  of  INITIATIVE 

15  f  of  INITIATIVE 


Justification  Section 

Total  Number  of  Index  Terms 
Used* 

Sum  of  Variables  31  through  59 

wf  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT* 

Sum  of  Variables  1  through  29* 
f  of  DRIVE* 

Total  Number  of  3  (New  1) 
Weights 

wf  of  STAFFING 
f  of  RESOURCEFULNESS 
wf  of  SERVICE  MOTIVATION 
f  of  REPUTE 

wf  of  ASSET  TO  THE  NAVY 
wf  of  FLEXIBILITY 
wf  of  GROOMING  AND  ATTIRE 
f  of  AWARDS  AND  PUNISHMENT 
wf  of  CONTROLLING 


Also  selected  by  the  stepwise  discriminant  analysis  program  for  this  section 
in  the  cross  validation  sample. 
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TABLE  33 

VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  CROSS  VALIDATION  AT's 


Step  Evaluation  Section 

1  f  of  COOPERATION* 

2  Total  Number  of  5  (New  3) 

Weights* 

3  f  of  TECHNICAL  SKILLS 

4  f  of  LEADERSHIP  AND  DIRECTING 

5  Total  Number  of  2  (New  -1) 

Weights* 

6  wf  of  RESPONSIVENESS 

7  f  of  RESOURCEFULNESS 

8  f  of  PRODUCTIVITY  AND  ACHIEVE¬ 

MENT* 

9  f  of  SERVICE  MOTIVATION 

10  wf  of  TECHNICAL  SKILLS 

11  Total  Number  of  4  (New  2) 

Weights 

12  f  of  AWARDS  AND  PUNISHMENT 

13  f  of  PROFESSIONALISM 

14  Total  Number  of  Index  Terms 

Used* 

15  Total  Number  of  Words  In  Text 


Justification  Section 

Total  Number  of  Index  Terms 
Used* 

wf  of  TECHNICAL  SKILLS 

wf  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 

wf  of  SKILLS  AND  ABILITIES 
f  of  COMMUNICATION 

wf  of  ORGANIZATION 
wf  of  ASSET  TO  THE  NAVY 
wf  of  REPUTE 

wf  of  AWARDS  AND  PUNISHMENT 
f  of  INTELLECTUAL  FUNCTIONING* 
wf  of  POTENTIAL 

f  of  REPRESENTATION 

Total  Number  of  Words  in  Text 

wf  of  RESPONSIVENESS 

f  of  RESPONSIVENESS 


Also  selected  by  the  stepwise  discriminant  analysis  program  for  this  section 
In  the  pilot  study  AT  subsample. 
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TABLE  34 

VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  PILOT  STUDY  AT’s 


Step  Evaluation  Section 

1  Total  Number  of  5  (New  3) 

Weights* 

2  Total  Number  of  2  (New  -1) 

Weights* 

3  wf  of  AWARDS  AND  PUNISHMENT 

4  f  of  COMMUNICATION 

5  f  of  PRODUCTIVITY  AND  ACHIEVE¬ 

MENT* 

6  f  of  COOPERATION* 

7  wf  of  MANAGEMENT  FUNCTIONS 

8  wf  of  PRODUCTIVITY  AND  ACHIEVE¬ 

MENT 

9  wf  of  SERVICE  MOTIVATION 

10  Total  Number  of  Index  Terms 

Used* 


Justification  Section 

Total  Number  of  Index  Terms 
Used* 

Total  Number  of  5  (New  3) 
Weights* 

f  of  CONDUCT,  INTEGRITY,  AND 
PRIDE 

f  of  INTELLECTUAL  FUNCTIONING* 
wf  of  INTELLECTUAL  FUNCTIONING 

wf  of  GROOMING  AND  ATTIRE 
f  of  TECHNICAL  SKILLS 
f  of  REPUTE 

wf  of  FLEXIBILITY 

f  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 


11  wf  of  USE  OF  COMMUNICATION 

12  f  of  USE  OF  COMMUNICATION 

13  wf  of  REPRESENTATION 

14  wf  of  GROOMING  AND  ATTIRE 

15  wf  of  DRIVE 


Sum  of  Variables  1  through  29 
wf  of  ENDURANCE 
wf  of  SERVICE  MOTIVATION 
wf  of  CONTROLLING 
f  of  ENDURANCE 


Also  selected  by  the  stepwise  discriminant  analysis  program  for  this  section 
in  the  cross  validation  AT  subsample. 
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denotes  variables  within  each  section  that  were  selected  in  both  samples.  On 
the  evaluation  section,  five  of  the  first  15  variables  selected  were  the  same 
for  the  cross  validation  AT’s  and  the  pilot  study  AT’s,  As  expected,  Total 
Number  of  5  (New  3)  Weights  and  Total  Number  of  2  (New  -1)  Weights  were  among 
these  five  variables.  On  the  Justification  section,  two  of  the  first  15  varia¬ 
bles  selected  were  the  same  for  the  cross  validation  AT’s  and  the  pilot  study 
AT’s.  Again,  Total  Number  of  Index  Terms  Used  was  the  first  variable  selected 
for  both  the  cross  validation  AT’s  and  the  pilot  study  AT’s. 

Tables  35  and  36  show  the  first  15  variables  selected  by  the  stepwise  dis¬ 
criminant  analysis  program  for  the  cross  validation  BT’s  and  the  pilot  study 
BT’s  in  the  evaluation  and  justification  sections.  Again,  an  asterisk  denotes 
variables  within  each  section  that  were  selected  in  both  samples.  On  the 
evaluation  section,  two  of  the  first  15  variables  selected  were  the  same  for 
the  cross  validation  BTfs  and  the  pilot  study  BT’s.  Total  Number  of  5  (New  3) 
Weights  was  the  first  variable  selected  for  both  the  cross  validation  BT’s  and 
the  pilot  study  BT’s.  On  the  Justification  section,  five  of  the  first  15  varia¬ 
bles  selected  were  the  same  for  the  cross  validation  BT’s  and  the  pilot  study 
BT’s.  As  with  the  two  AT  subsamples,  Total  Number  of  Index  Terms  Used  was  the 
first  variable  selected  for  both  the  cross  validation  BT’s  and  the  pilot  study 
BT’s. 


The  key  discriminating  variables  for  the  cross  validation  and  pilot  study 
samples  as  well  as  for  the  AT  and  BT  subsamples  were  Total  Number  of  5  (New  3) 
Weights  and  Total  Number  of  2  (New  -1)  Weights  in  the  evaluation  section.  In 
the  justification  section  without  exception  the  key  discriminating  variable 
was  Total  Number  of  Index  Terms  Used,  What  are  the  results  like  for  the  gener¬ 
alization  sample?  Table  37  shows  that  these  same  three  variables  were  those 
selected  first  in  the  stepwise  discriminant  analysis  of  the  generalization  sam¬ 
ple.  Total  Number  of  2  (New  -1)  Weights  and  Total  Number  of  5  (New  3)  Weights 
were  selected  first  and  second  in  the  evaluation  section.  Total  Number  of  In¬ 
dex  Terms  Used  was  selected  first  in  the  Justification  section. 

When  the  two  occupational  groups  comprising  the  generalization  sample  are 
considered  individually,  the  results  are  not  quite  unanimous.  Neither  Total 
Number  of  5  (New  3)  Weights  nor  Total  Number  of  2  (New  -1)  Weights  was  select¬ 
ed  in  the  evaluation  section  for  the  generalization  CS’s  (see  Table  38).  How¬ 
ever,  Total  Number  of  Index  Terms  Used  was  the  first  variable  selected  in  the 
Justification  section  for  the  generalization  CS’s.  In  Table  39  showing  the 
results  for  the  generalization  RM’s,  Total  Number  of  2  (New  -1)  Weights  was 
selected  as  the  first  variable  in  the  evaluation  section  and  Total  Number  of  5 
(New  3)  Weights  was  selected  as  the  fifth  variable.  Again,  Total  Number  of 
Index  Terms  Used  was  selected  first  in  the  Justification  section. 

Two  findings  are  striking.  Without  exception  better  classification  was 
achieved  in  the  content  analysis  of  the  narrative  comments  in  the  Justifica¬ 
tion  section,  and  without  exception  the  first  variable  selected  in  the  various 
stepwise  discriminant  analyses  for  the  Justification  section  was  Total  Number 
of  Index  Terms  Used.  This  variable  reflects  the  variety  of  specific  areas  of 
a  ratee’s  performance  that  the  evaluator  chose  to  comment  on,  and  is  measured 
by  the  number  of  different  index  terms  selected  by  the  indexer  to  encompass 
the  narrative  content.  It  appears  that  the  range  of  skills  and  abilities  that 
a  chief  petty  officer  possesses  may  be  a  key  factor  in  his  superior  performance. 
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TABLE  35 

VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  CROSS  VALIDATION  BT's 


Step 

Evaluation  Section 

Justification  Section 

1 

Total  Number  of  5  (New  3) 

Weights* 

Total  Number  of  Index  Terms 
Used* 

2 

Total  Number  of  Index  Terms 

Used 

wf  of  LEADERSHIP  AND  DIRECTING 

3 

Sum  of  Variables  31  through  59 

Sum  of  Variables  1  through  29 

4 

wf  of  RESOURCEFULNESS 

f  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT* 

5 

f  of  COOPERATION 

wf  of  CONDUCT,  INTEGRITY,  AND 
PRIDE 

6 

f  of  RESOURCEFULNESS 

f  of  DRIVE 

7 

Total  Number  of  Words  in  Text 

wf  of  SERVICE  MOTIVATION 

8 

f  of  AWARDS  AND  PUNISHMENT 

f  of  INITIATIVE* 

9 

f  of  ORGANIZATION 

f  of  RESPONSIVENESS 

10 

f  of  TECHNICAL  SKILLS 

f  of  ASSET  TO  THE  NAVY* 

11 

f  of  ASSET  TO  THE  NAVY 

wf  of  DRIVE 

12 

wf  of  ASSET  TO  THE  NAVY* 

wf  of  RESOURCEFULNESS 

13 

wf  of  FLEXIBILITY 

f  of  RESOURCEFULNESS* 

14 

wf  of  REPUTE 

f  of  PROFESSIONALISM 

15 

f  of  STAFFING 

f  of  REPRESENTATION 

* 

Also  selected  by  the  stepwise  discriminant  analysis  program  for  this  sectic 
in  the  pilot  study  BT  subsample. 

75 


TABLE  36 

VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  PILOT  STUDY  BT’s 


Step  Evaluation  Section 

1  Total  Number  of  5  (New  3) 

Weights 

2  Total  Number  of  2  (New  -1) 

Weights 

3  f  of  LEADERSHIP  AND  DIRECTING 

4  wf  of  PROFESSIONALISM 

5  f  of  RESPONSIVENESS 

6  wf  of  GROOMING  AND  ATTIRE 

7  wf  of  TECHNICAL  SKILLS 

8  wf  of  CONTROLLING 

9  f  of  SKILLS  AND  ABILITIES 

10  wf  of  RELIABILITY  AND  DEPENDA¬ 

BILITY 

11  wf  of  MANAGEMENT  FUNCTIONS 

12  wf  of  USE  OF  COMMUNICATION 

13  wf  of  ASSET  TO  THE  NAVY* 

14  f  of  MANAGEMENT  FUNCTIONS 

15  wf  of  STAFFING 


Justification  Section 

Total  Number  of  Index  Terms 
Used* 

wf  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 

f  of  AWARDS  AND  PUNISHMENT 

Sum  of  Variables  31  through  59 

f  of  REPUTE 

f  of  RESOURCEFULNESS* 

f  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT* 

wf  of  CONTROLLING 
f  of  SERVICE  MOTIVATION 
f  of  ASSET  TO  THE  NAVY* 

f  of  ENDURANCE 

Total  Number  of  3  (New  1) 
Weights 

wf  of  REPUTE 

f  of  USE  OF  COMMUNICATION 
f  of  INITIATIVE* 


Also  selected  by  the  stepwise  discriminant  analysis  program  for  this  section 
in  the  cross  validation  BT  subsample. 
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TABLE  37 

VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  GENERALIZATION  SAMPLE 


Step  Evaluation  Section 

1  Total  Number  of  2  (New  -1) 

Weights 

2  Total  Number  of  5  (New  3) 

Weights 

3  wf  of  POTENTIAL 

4  f  of  COMMUNICATION 

5  wf  of  MANAGEMENT  FUNCTIONS 

6  f  of  RELIABILITY  AND  DEPENDA¬ 

BILITY 

7  f  of  INTELLECTUAL  FUNCTIONING 

8  wf  of  SERVICE  MOTIVATION 

9  f  of  ORGANIZATION 

10  f  of  RESPONSIVENESS 

11  wf  of  INITIATIVE 

12  wf  of  PLANNING 

13  wf  of  RELIABILITY  AND  DEPENDA¬ 

BILITY 

14  f  of  REPRESENTATION 

15  wf  of  STAFFING 


Justification  Section 

Total  Number  of  Index  Terms 
Used 

f  of  COOPERATION 

wf  of  STAFFING 
wf  of  COMMUNICATION 
wf  of  PLANNING 
wf  of  POTENTIAL 

f  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 

wf  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 

Sum  of  Variables  31  through  59 
wf  of  GROOMING  AND  ATTIRE 
f  of  ENDURANCE 
f  of  REPUTE 

wf  of  INTELLECTUAL  FUNCTIONING 

f  of  RESPONSIVENESS 
wf  of  REPUTE 


Step 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 
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TABLE  38 

VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  GENERALIZATION  CS's 


Evaluation  Section 

Justification  Section 

wf  of  CONTROLLING 

Total  Number  of  Index  Terms 
Used 

wf  of  ASSET  TO  THE  NAVY 

f  of  PROFESSIONALISM 

f  of  MANAGEMENT  FUNCTIONS 

f  of  INITIATIVE 

wf  of  SKILLS  AND  ABILITIES 

wf  of  COMMUNICATION 

Total  Number  of  Index  Terms 

Used 

wf  of  REPRESENTATION 

f  of  ENDURANCE 

wf  of  COOPERATION 

Sum  of  Variables  31  through  59 

f  of  CONTROLLING 

wf  of  ORGANIZATION 

f  of  POTENTIAL 

wf  of  POTENTIAL 

f  of  PRODUCTIVITY  AND  ACHIEVE 
MENT 

wf  of  PLANNING 

wf  of  SKILLS  AND  ABILITIES 

f  of  SERVICE  MOTIVATION 

wf  of  PLANNING 

f  of  PLANNING 

Total  Number  of  Words  In  Text 

Total  Number  of  3  (New  1) 

Weights 

wf  of  PROFESSIONALISM 

f  of  LEADERSHIP  AND  DIRECTING 

f  of  SKILLS  AND  ABILITIES 

f  of  INITIATIVE 

f  of  RELIABILITY  AND  DEPENDA¬ 
BILITY 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 


TABLE  39 


VARIABLES  SELECTED  BY  THE  STEPWISE  DISCRIMINANT  ANALYSIS  PROGRAM 
AT  STEPS  1  THROUGH  15  FOR  THE  GENERALIZATION  RM's 


Evaluation  Section 

Justification  Section 

Total  Number  of  2  (New  -1) 
Weights 

Total  Number  of  Index  Terms 
Used 

wf  of  AWARDS  AND  PUNISHMENT 

Sum  of  Variables  31  through  59 

wf  of  POTENTIAL 

wf  of  PRODUCTIVITY  AND  ACHIEVE 
MENT 

wf  of  MANAGEMENT  FUNCTIONS 

f  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 

Total  Number  of  5  (New  3) 
Weights 

f  of  COOPERATION 

f  of  RELIABILITY  AND  DEPENDA¬ 
BILITY 

wf  of  GROOMING  AND  ATTIRE 

f  of  COMMUNICATION 

wf  of  STAFFING 

f  of  POTENTIAL 

wf  of  ENDURANCE 

wf  of  REPUTE 

f  of  STAFFING 

f  of  REPUTE 

wf  of  PLANNING 

f  of  INTELLECTUAL  FUNCTIONING 

Total  Number  of  4  (New  2) 
Weights 

f  of  RESPONSIVENESS 

f  of  POTENTIAL 

wf  of  RESPONSIVENESS 

f  of  INITIATIVE 

wf  of  SERVICE  MOTIVATION 

wf  of  TECHNICAL  SKILLS 

wf  of  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT 


f  of  REPUTE 
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Another  important  finding  is  that  better  classification  was  achieved  when 
each  of  the  four  occupational  ratings  studied  in  this  research  project  was  con¬ 
sidered  individually.  In  addition  to  Total  Number  of  5  (New  3)  Weights,  Total 
Number  of  2  (New  -1)  Weights,  and  Total  Number  of  Index  Terms  Used,  a  number 
of  specific  index  terms  among  the  first  five  selected  should  be  called  out  as 
potential  key  variables  in  discriminating  between  superior  chief  petty  offi¬ 
cers  and  their  slightly  less  qualified  colleagues.  For  the  cross  validation 
AT' s  these  variables  were  COMMUNICATION;  COOPERATION;  LEADERSHIP  AND  DIRECTING; 
PRODUCTIVITY  AND  ACHIEVEMENT;  TECHNICAL  SKILLS;  and  SKILLS  AND  ABILITIES.  For 
the  pilot  study  AT's  these  variables  were  AWARDS  AND  PUNISHMENT;  COMMUNICATION; 
CONDUCT,  INTEGRITY,  AND  PRIDE;  INTELLECTUAL  FUNCTIONING;  and  PRODUCTIVITY  AND 
ACHIEVEMENT.  The  potential  key  variables  for  the  cross  validation  BT's  were 
CONDUCT,  INTEGRITY,  AND  PRIDE;  COOPERATION;  LEADERSHIP  AND  DIRECTING;  PRODUC¬ 
TIVITY  AND  ACHIEVEMENT;  and  RESOURCEFULNESS.  For  the  pilot  study  BT's  these 
variables  were  AWARDS  AND  PUNISHMENT;  LEADERSHIP  AND  DIRECTING;  PRODUCTIVITY 
AND  ACHIEVEMENT;  PROFESSIONALISM;  REPUTE;  and  RESPONSIVENESS.  The  potential 
key  variables  for  the  CS's  were  ASSET  TO  THE  NAVY;  COMMUNICATION;  CONTROLLING; 
INITIATIVE;  MANAGEMENT  FUNCTIONS;  PROFESSIONALISM;  REPRESENTATION;  and  SKILLS 
AND  ABILITIES.  For  the  RM's  these  variables  were  AWARDS  AND  PUNISHMENT;  COOP¬ 
ERATION;  MANAGEMENT  FUNCTIONS;  POTENTIAL;  and  PRODUCTIVITY  AND  ACHIEVEMENT. 

These  same  specific  index  terms  were  also  the  variables  achieving  statistical 
significance  beyond  the  .001  level  of  probability  in  the  Mann-Whitney  U  test 
as  well  as  the  t  test  of  mean  difference  for  the  comparison  on  the  cross  vali¬ 
dation  and  generalization  samples  between  the  Middle  and  Upper  criterion  groups, 
the  most  difficult  discrimination  to  be  made  between  any  two  of  the  three  cri¬ 
terion  groups  (see  Tables  20  to  23). 

Ten  of  the  29  index  terms  do  not  appear  to  lend  as  much  assistance  in  the 
discrimination  task  as  the  key  variables  cited  above.  These  less  useful  terms 
are  the  following:  DRIVE;  ENDURANCE;  FLEXIBILITY;  GROOMING  AND  ATTIRE;  ORGANI¬ 
ZATION;  PLANNING;  RELIABILITY  AND  DEPENDABILITY;  SERVICE  MOTIVATION;  STAFFING; 
and  USE  OF  COMMUNICATION.  However,  all  but  two  of  these  ten  terms  were  select¬ 
ed  as  one  of  the  sixth  to  the  tenth  variables  in  at  least  one  of  the  stepwise 
discriminant  analyses  performed.  The  two  exceptions  were  FLEXIBILITY  and  USE 
OF  COMMUNICATION,  the  least  useful  terms  used  in  this  study.  These  findings 
suggest  that  a  smaller  number  of  dimensions  than  the  full  complement  of  67 
quantitative  variables  derived  from  the  indexing  procedure  can  be  used  to  iden¬ 
tify  superlative  CPO's  whose  superior  performance  recommends  them  as  candidates 
for  promotion  to  a  higher  level  of  responsibility. 
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SECTION  5.  RELIABILITY  STUDY 

A  comprehensive  reliability  study  was  conducted  whose  objectives  were  two¬ 
fold:  (1)  to  determine  the  level  of  agreement  among  several  individuals  all 

of  whom  independently  would  perform  a  content  analysis  of  the  same  corpus  of 
Evaluation  Reports,  and  (2)  to  investigate  if  nonresearchers  could  be  trained 
successfully  to  apply  the  complex  content  analysis  methodology  developed  in 
the  pilot  study. 

A  set  of  48  Evaluation  Reports  was  selected  by  the  Navy  Personnel  Research 
and  Development  Center,  representing  a  cross  section  of  the  kinds  of  reports 
included  in  the  overall  experimental  design  for  the  cross  validation  and  gen¬ 
eralization  samples.  In  each  of  these  48  Evaluation  Reports  the  evaluation 
section  was  separated  from  the  justification  section  so  that  the  narrative  com¬ 
ments  for  each  section  were  not  considered  together.  This  resulted  in  a  group 
of  96  randomized  pieces  of  narrative  text  to  be  indexed  in  the  reliability 
study.  To  each  of  these  96  pieces  of  narrative  text  was  appended  the  corre¬ 
sponding  sections  4A  and  4B  of  the  Evaluation  Report  form.  These  two  sections 
provide  a  description  of  the  ratee’s  primary  and  collateral  duties  and  should 
be  read  as  background  information  before  beginning  to  index  the  narrative  text. 
Each  of  these  96  minidocuments  was  assigned  a  4-digit  identification  number  by 
NPRDC,  Xeroxed  in  multiple  copies,  and  sent  to  R-K  Research  and  System  Design 
to  be  used  as  the  data  base  in  the  reliability  study. 

Four  individuals  participated  in  the  reliability  study:  (1)  the  experi¬ 
enced  indexer  who  also  indexed  the  pilot  study  sample,  the  cross  validation 
sample,  and  the  generalization  sample;  (2)  the  principal  investigator;  (3) 
an  inexperienced  indexer  (inexperienced  indexer  A)  with  two  years  of  college 
in  the  liberal  arts;  and  (4)  another  inexperienced  indexer  (inexperienced  in¬ 
dexer  B)  with  executive  secretary  experience.  To  this  end  a  training  manual 
was  prepared  by  the  experienced  indexer  and  the  principal  investigator  to 
assist  the  two  neophyte  indexers  in  understanding  their  assignment.  The  train¬ 
ing  manual  in  its  entirety  is  included  in  this  report  as  Appendix  A.  The  ver¬ 
sion  of  the  training  manual  shown  in  Appendix  A  is  not  the  original  version 
that  was  used  to  train  the  four  reliability  indexers,  but  rather  is  an  updated 
version  that  includes  voluminous  examples  of  how  to  handle  difficult  indexing 
decisions  and  which  also  attempt  to  eliminate  areas  of  confusion  brought  to 
light  in  analyzing  the  results  of  the  reliability  study. 

Six  intensive  training  sessions  were  conducted  by  the  experienced  indexer 
in  order  to  try  to  bring  all  four  indexers  up  to  a  common  level  of  expertise 
before  beginning  the  actual  study.  Obviously,  this  objective  could  only  be 
met  partially  in  view  of  the  varying  educational  backgrounds  of  the  four  relia¬ 
bility  indexers  and  their  different  levels  of  previous  exposure  to  the  indexing 
dictionary.  The  study  itself  proved  to  be  a  traumatic  experience  for  the  three 
reliability  indexers  who  had  not  spent  the  past  year  doing  the  actual  indexing 
of  the  cross  validation  and  generalization  samples,  probably  because  they  were 
still  in  the  early  stages  of  their  learning  curves.  The  task  given  to  each 
participant  was  extremely  difficult  and  can  be  likened  to  a  take-home,  open- 
book  final  examination  without  a  time  limit.  However,  given  the  experience  of 
participating  in  the  study,  all  four  of  the  reliability  indexers  now  feel  that 
they  are  better  qualified  to  function  as  regular  indexers  and  could  perform 
this  assignment  in  a  consistent  manner. 
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When  all  four  reliability  indexers  had  completed  indexing  the  96  pieces 
of  narrative  text,  their  indexing  decisions  were  recorded  side  by  side  on 
work  sheets  for  each  segment  of  narrative  text  indexed.  These  work  sheets 
provided  the  data  base  for  computing  agreement  statistics.  In  all  of  the  sta¬ 
tistical  computations  reported  subsequently  in  this  section,  assignment  of 
the  index  terms  was  considered  to  be  a  separate  intellectual  task  from  assign¬ 
ing  the  corresponding  weights  based  on  the  modifying  adjectives  and  adverbs. 
There  is  good  justification  for  analyzing  the  reliability  study  results  in 
these  two  contexts.  When  an  indexer  studied  a  segment  of  narrative  text,  the 
first  step  was  to  select  an  appropriate  index  term  or  terms  from  among  the  29 
possibilities  that  best  described  the  substantive  content  of  the  text.  Once 
the  indexer  had  completed  this  first  phase  of  the  content  analysis,  then  the 
segment  of  narrative  text  was  rescanned  to  identify  the  adjectives  and  adverbs 
that  defined  the  numerical  weight  to  be  assigned  to  each  index  term  chosen. 
Considering  these  judgments  as  two  sequential  decision  processes  also  made 
the  results  of  the  reliability  study  more  amenable  to  statistical  analysis  as 
will  be  shown  in  the  subsequent  discussion. 

As  early  as  1960  Cohen,  in  introducing  a  new  agreement  statistic  called 
kappa,  pointed  out  that  for  most  problems  in  nominal  scale  agreement  between 
two  judges  or  decision  makers,  many  investigators  compute  a  contingency  chi 
square  as  a  test  of  the  hypothesis  of  chance  agreement,  and  some  investigators 
have  gone  on  to  compute  the  contingency  coefficient,  C,  as  a  measure  of  degree 
of  agreement. However,  Cohen  concluded  that  the  use  of  chi  square  (x2)  ,  and 
therefore,  the  C  which  is  based  on  it  for  the  evaluation  of  agreement  is  inde¬ 
fensible.  When  applied  to  a  contingency  table,  x2  tests  the  null  hypothesis 
with  regard  to  association ,  not  agreement.  Therefore,  x2  and  C  are  inappro¬ 
priate  statistics  for  measuring  agreement  since  they  will  be  inflated  quite 
impartially  by  any  departure  from  chance  association,  either  disagreement  or 
agreement .  In  order  to  remedy  this  situation,  Cohen  suggested  a  new  coeffi¬ 
cient,  kappa,  to  measure  the  degree  of  agreement  in  nominal  scales,  and  to 
provide  means  for  testing  hypotheses  and  setting  confidence  limits  for  this 
coef f ic ient . 

Quoting  from  Cohen’s  1960  article  [20,  pp.  39-40],  "...for  any  problem  in 
nominal  scale  agreement  between  two  judges,  there  are  only  two  relevant  quan¬ 
tities: 

p •  =  the  proportion  of  units  in  which  the  judges  agreed 

Pc  83  the  proportion  of  units  for  which  agreement  is  expected  by  chance. 

The  test  of  agreement  comes  then  with  regard  to  the  1  -  p  of  the  units 
for  which  the  hypothesis  of  no  association  would  predict  disagreement  between 
the  judges.  This  term  will  serve  as  the  denominator. 

uTo  the  extent  to  which  nonchance  factors  are  operating  in  the  direction 
of  agreement,  p  will  exceed  p  ;  their  difference,  p  -  p  ,  represents  the  pro¬ 
portion  of  the  cases  in  which  Heyond-chance  agreemen?  occurred  and  is  the  nu¬ 
merator  of  the  coefficient. 

"The  coefficient  k  is  simply  the  proportion  of  chance-expected  disagree¬ 
ments  which  do  not  occur,  or  alternatively,  it  is  the  proportion  of  agreement 
after  chance  agreement  is  removed  from  consideration: 
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P  -  p 

o  c 

1  -  2> 


The  significance  of  an  obtained  k  is  determined  by  dividing  <  by  a 


where  a 


N(1  -  Pc) 


The  resulting  critical  ratio  is  referred  to  the 


normal  curve.  However,  Cohen  has  pointed  out  that  it  is  generally  of  as  little 
value  to  test  k  for  significance  as  it  is  for  any  other  reliability  coefficient 

- to  know  merely  that  k  is  beyond  chance  is  trivial  since  one  usually  expects 

much  more  than  this  in  the  way  of  reliability  in  psychological  measurement. 
However,  the  size  of  the  critical  ratio  does  provide  some  immediate  feedback 
concerning  the  magnitude  of  the  agreement  achieved  beyond  the  level  expected 
by  chance.  Probably  a  more  useful  way  to  interpret  the  significance  of  an 
obtained  k  is  in  terms  of  the  maximum  value  of  k.  The  theoretical  upper  limit 
of  k  is  +1.00,  but  this  limit  can  only  be  reached  if  the  off-diagonal  (dis¬ 
agreement)  cells  in  the  agreement  matrix  are  all  zero.  This  in  turn  demands 
that  the  marginal  probabilities  for  each  diagonal  (agreement)  cell  must  be 
identical.  Perfect  agreement  between  two  judges  is  rarely  achieved,  and  there¬ 
fore,  the  marginal  distributions  in  any  agreement  matrix  are  not  identical. 

This  means  that  in  practice  the  upper  limit  of  k  is  never  +1.00  but  rather  some 
lesser  value.  The  maximum  value  of  k  is  set  by  the  marginal  distributions  in 
any  particular  application  of  the  kappa  agreement  statistic,  and  it  can  be  cal¬ 
culated.  A  comparison  of  the  obtained  k  with  its  maximum  upper  limit  computed 
from  the  marginal  distributions  provides  the  investigator  with  a  more  useful 
index  of  how  closely  the  agreement  level  that  was  achieved  between  two  judges 
approached  the  maximum  level  of  agreement  that  was  possible. 


The  kappa  statistic  was  the  measure  of  agreement  used  in  analyzing  the  in¬ 
dex  terms  assigned  by  the  four  reliability  indexers.  For  each  segment  of  nar¬ 
rative  text,  each  indexer  chose  a  term  or  terms  from  the  list  of  29  possibili¬ 
ties,  or  the  decision  was  made  that  no  term  should  be  used.  From  a  careful 
analysis  of  these  indexing  decisions  for  each  pair  of  reliability  indexers,  six 
pairwise  agreement  matrices  were  constructed.  These  were  30  by  30  matrices, 
with  the  29  index  terms  representing  29  of  the  30  nominal  categories  and  No  In¬ 
dex  Term  Used  representing  the  30th  nominal  category.  The  pairwise  indexing 
decisions  for  each  segment  of  narrative  text  analyzed  across  all  96  documents 
in  the  reliability  study  data  base  were  tabulated  into  the  appropriate  cell  of 
the  agreement  matrix  for  the  particular  pair  of  indexers  being  compared.  The 
30  diagonal  cells  of  the  agreement  matrix  denote  agreement  between  the  two  in¬ 
dexers  in  assigning  index  terms;  all  of  the  off-diagonal  elements  in  the  matrix 
represent  instances  in  which  the  two  indexers  disagreed  in  their  selection  of 
terms.  The  total  number  of  entries  in  these  six  matrices  varied  slightly  among 
the  six  pairwise  comparisons  between  the  four  reliability  indexers,  but  in  all 
instances  they  were  very  large,  ranging  from  1,230  tallies  to  1,389  tallies. 
Consequently,  the  size  of  the  reliability  study  data  base  can  be  considered  to 
be  large  enough  to  provide  a  stable  measure  of  the  level  of  agreement  achieved 
in  performing  this  complex  intellectual  task. 

Table  40  shows  the  results  of  the  kappa  analysis  of  the  six  pairwise  com¬ 
parisons  between  the  four  reliability  indexers  in  selecting  index  terms  for 
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TABLE  40 


RESULTS  OF  THE  KAPPA  ANALYSIS  FOR  THE  SIX  PAIRWISE  COMPARISONS 
BETWEEN  THE  FOUR  RELIABILITY  INDEXERS  IN  SELECTING  INDEX  TERMS 
FOR  THE  ENTIRE  RELIABILITY  STUDY  DATA  BASE 


Pairwise  Comparisons 

Between  Each  Pair  of 
Reliability  Indexers 

K 

The  experienced  indexer  vs. 
the  principal  investigator 

.73 

The  experienced  indexer  vs. 
inexperienced  indexer  A 

.88 

The  experienced  indexer  vs. 
inexperienced  indexer  B 

.72 

The  principal  investigator 
vs.  inexperienced  indexer  A 

.73 

The  principal  investigator 
vs.  inexperienced  indexer  B 

.71 

Inexperienced  indexer  A  vs. 
inexperienced  indexer  B 

.78 

O 

o 

* 

z 

K 

max 

.0065 

111.82 

.90 

.0072 

123.49 

.97 

.0068 

106.24 

.90 

.0065 

111.82 

.89 

.0064 

111.06 

.90 

.0068 

115.56 

.92 

A  z  of  3.29  is  significant  at  the  *001  level  of  probability.  Therefore,  all 
of  the  z  values  reported  in  this  table  are  extremely  significant  and  lead  to 
rejection  of  the  null  hypothesis  that  the  obtained  k  does  not  exceed  the 
chance  level  of  agreement. 


the  entire  reliability  study  data  base.  The  second  column  in  this  table  shows 
the  value  of  k;  the  third  column  shows  the  standard  error  of  k;  and  the  fourth 
column  lists  the  normal  deviate,  z,  obtained  by  dividing  k  by  its  standard  er¬ 
ror.  All  of  the  z  values  are  very  large,  and  consequently,  extremely  signifi¬ 
cant,  indicating  that  in  all  six  comparisons  the  null  hypothesis  that  the  ob¬ 
tained  k  does  not  exceed  the  chance  level  of  agreement  can  be  rejected.  The 
last  column  in  Table  40  provides  the  maximum  possible  value  of  kappa  for  each 
of  the  six  pairwise  comparisons.  These  values  can  be  used  as  an  upper  limit 
for  comparing  the  level  of  agreement  actually  achieved  with  the  maximum  level 
possible  given  the  marginal  distributions.  Thus,  in  the  first  comparison  be¬ 
tween  the  experienced  indexer  and  the  principal  investigator,  the  k  obtained 
was  .73  compared  to  a  possible  maximum  value  of  .90.  The  best  agreement  in 
selecting  index  terms  was  obtained  between  the  experienced  indexer  and  inex¬ 
perienced  indexer  A,  a  k  of  .88  where  the  maximum  k  possible  in  this  instance 
was  .97.  This  is  a  heartening  finding,  suggesting  that  an  individual  without 
a  research  background  in  only  six  training  sessions  can  be  trained  to  apply 
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the  complex  content  analysis  methodology  developed  in  this  research  project. 

With  additional  training  and  further  experience,  this  individual  could  be  ex¬ 
pected  to  apply  these  indexing  skills  in  an  even  more  consistent  manner. 

The  other  values  of  k  in  Table  40  are  not  as  large  as  the  one  for  the  com¬ 
parison  between  the  experienced  indexer  and  inexperienced  indexer  A.  However, 
they  all  range  between  .71  and  .78,  with  .71  probably  representing  the  lower 
limit  of  reliability  achievable  in  a  study  of  this  kind.  With  additional 
training  in  those  areas  where  there  was  confusion  in  the  minds  of  the  relia¬ 
bility  indexers  as  they  wrestled  with  the  task  of  selecting  the  most  appropri¬ 
ate  index  term,  the  expectation  would  be  that  better  agreement  could  be 
achieved  among  these  same  four  individuals  if  they  were  to  replicate  this  ex¬ 
periment.  All  four  reliability  indexers  have  expressed  their  concordance  with 
this  expectation. 

A  careful  perusal  of  the  six  agreement  matrices  from  which  the  statistics 
presented  in  Table  40  were  derived  was  very  instructive.  Most  of  the  off- 
diagonal  cells  were  empty.  When  there  were  tallies,  they  were  sporadic  and 
scattered  with  only  one  or  two  tallies  appearing  in  an  occasional  cell  off  the 
diagonal.  However,  two  areas  of  confusion  were  prominently  displayed  in  these 
six  matrices.  One  minor  area  of  confusion  was  between  PRODUCTIVITY  AND  ACHIEVE¬ 
MENT  and  MANAGEMENT  FUNCTIONS,  although  this  confusion  could  not  be  considered 
to  be  extensive.  The  revision  of  the  training  manual  included  in  Appendix  A 
has  attempted  to  clarify  the  points  of  confusion  between  these  two  index  terms. 

The  other  area  of  confusion  is  major  and  deserves  special  comment.  In  the 
six  training  sessions  it  was  pointed  out  that  certain  statements  in  the  narra¬ 
tive  text  describe  the  job  duties  and  the  qualifications  for  the  position  that 
the  ratee  occupies  rather  than  the  ratee’s  actual  qualifications  for  and  per¬ 
formance  in  this  position.  When  such  a  factual  statement  of  the  requirements 
for  the  position  were  included  in  the  narrative  text,  it  was  not  to  be  indexed 
since  it  told  nothing  about  the  ratee’s  qualifications  and  performance  per  se. 
Therefore,  it  was  not  considered  to  be  an  evaluative  statement.  Unambiguous 
examples  of  this  type  of  narrative  statement  taken  from  the  training  material 
are  the  following: 

Example  I.  As  the  Quality  Control  Chief  he  is  responsible  for  the 
continuous  updating  of  a  number  of  SOP’s  as  well  as  implement¬ 
ing  the  new  ones  that  are  required. 

Example  2.  Chief  XX  is  presently  serving  in  an  RM1  billet.  This 
is  because  he  made  Chief  Petty  Officer  in  November  1970. 

However,  when  confronted  with  indexing  the  reliability  study  data  base 
itself,  this  indexing  convention  was  frequently  misconstrued  by  the  three  re¬ 
liability  indexers  who  had  not  had  the  extensive  indexing  experience  that  the 
experienced  indexer  had  had.  The  following  two  examples  illustrate  where  the 
confusion  arose. 

Example  3.  Ratee fs  assignment  demands  particularly  delicate  tact, 
due  to  his  working  among  civilians  upon  whom  he  must  depend 
for  cooperation. 


86 


The  experienced  indexer  did  not  assign  any  index  terms  to  this  sentence,  con¬ 
cluding  that  it  was  a  statement  defining  the  requirements  of  the  job  position 
rather  than  how  the  ratee  performed  in  the  job.  All  of  the  other  three  relia¬ 
bility  indexers  inferred  that  the  individual  evaluating  the  ratee  was  actually 
referring  to  the  fact  that  the  ratee  possessed  tact  in  interfacing  with  the 
civilian  community.  The  principal  investigator  used  the  terms  CONDUCT,  INTEG¬ 
RITY,  AND  PRIDE  and  REPRESENTATION  to  index  this  sentence,  as  did  inexperienced 
indexer  B.  Inexperienced  indexer  A  used  only  CONDUCT,  INTEGRITY,  AND  PRIDE. 
There  was  substantial  agreement  among  the  three  less  experienced  reliability 
indexers,  but  none  of  them  matched  the  indexing  decision  of  the  experienced  in¬ 
dexer  that  they  were  trying  to  emulate.  Another  example  of  this  type  of  dis¬ 
agreement  is  the  following: 

Example  4.  Additionally,  he  must  supply  satisfactory  solutions  to 
the  many  problems  of  the  Company  Commanders  in  connection  with 
these  services. 

Again  the  experienced  indexer  considered  this  comment  to  be  a  statement  defin¬ 
ing  the  requirements  of  the  job  position  whereas  the  other  three  reliability 
indexers  attributed  the  skill  required  to  the  ratee.  The  principal  investiga¬ 
tor  and  inexperienced  indexer  B  called  the  skill  CONTROLLING;  inexperienced  in¬ 
dexer  A  called  it  PLANNING. 

Disagreement  in  which  the  experienced  indexer  did  not  assign  any  index 
terms  and  the  other  three  reliability  indexers  did  assign  one  or  more  terms  is 
very  noticeable  in  studying  the  three  agreement  matrices  in  which  the  three 
less  experienced  reliability  indexers  are  compared  with  the  experienced  index¬ 
er.  This  type  of  disagreement  also  occurred  in  the  three  comparisons  among  the 
less  experienced  indexers,  emphasizing  the  general  confusion  that  existed  in 
how  to  handle  statements  of  the  type  shown  in  Examples  3  and  4  above.  Addi¬ 
tional  training  aimed  at  clarifying  this  area  of  confusion  most  likely  would 
markedly  reduce  this  type  of  disagreement  and  raise  the  magnitude  of  k. 

Analysis  of  the  level  of  agreement  among  the  four  reliability  indexers  in 
assigning  numerical  weights  to  each  index  term  selected,  based  on  the  modify¬ 
ing  adjectives  and  adverbs,  was  performed  differently  than  the  analysis  of  the 
level  of  agreement  in  selecting  the  index  terms  themselves.  Selection  of  the 
index  terms  in  this  reliability  study  constituted  a  nominal  scale  whereas  as¬ 
signment  of  a  numerical  weight  to  each  index  term  selected  was  an  indexing  de¬ 
cision  involving  an  ordinal  scale.  Therefore,  more  powerful  agreement  statis¬ 
tics  could  be  employed.  Since  numerical  weights  on  a  scale  from  1  to  5  (New 
-2  to  New  3)  were  assigned  to  each  index  term  selected,  it  was  possible  to  com¬ 
pute  a  product  moment  correlation  coefficient  between  each  pair  of  reliability 
indexers.  The  new  transformed  weights  were  used  in  these  computations  since 
this  ordinal  scale  provided  a  more  justifiable  way  of  measuring  the  situation 
in  which  one  indexer  did  not  select  an  index  term  but  the  other  indexer  did 
(see  Table  9). 

In  addition  to  computing  these  six  product  moment  correlation  coefficients, 
another  agreement  statistic,  weighted  kappa,  was  also  calculated  in  order  to 
determine  if  it  agreed  with  the  results  of  the  correlational  analysis.  In  1968 
Cohen  published  another  article  generalizing  the  kappa  statistic  to  the  situa¬ 
tion  in  which  disagreements  of  varying  gravity  can  be  weighted  accordingly. 
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Application  of  weighted  kappa  to  quantifying  the  level  of  agreement  in  psychi¬ 
atric  diagnosis  was  also  shown  by  Cohen  and  his  colleagues.  2 


Weighted  kappa  is  an  agreement  statistic  corrected  for  chance  agreement, 
to  be  used  when  different  kinds  of  disagreement  are  to  be  differentially  weight¬ 
ed  in  the  agreement  index.  The  desired  weighting  is  accomplished  by  an  a 
priori  assignment  of  weights  to  the  r  by  c  cells  of  the  agreement  matrix,  and 
must  be  done  very  carefully  because  the  weights  assigned  are  an  integral  part 
of  how  agreement  is  defined,  and  therefore,  how  it  is  measured  with  weighted 
kappa  (k  ) .  Table  41  shows  the  weighting  algorithm  that  was  used  in  computing 
k  for  assessing  the  level  of  agreement  in  assigning  numerical  weights,  based 
on  the  modifying  adjectives  and  adverbs,  to  the  index  terms  selected  in  the 
reliability  study.  The  first  step  in  computing  k  was  to  construct  a  6  by  6 
agreement  matrix  between  each  pair  of  reliabilityxindexers  that  encompassed 
all  of  the  pairwise  numerical  weights  that  were  assigned  to  index  terms  based 
on  their  modifying  adjectives  and  adverbs.  These  numerical  weights  were  tabu¬ 
lated  in  the  agreement  matrix  across  all  96  documents  in  the  reliability  study 
data  base.  Using  the  first  row  of  Table  41  as  an  example,  if  Reliability  In¬ 
dexer  I  and  Reliability  Indexer  II  both  had  assigned  a  numerical  weight  of  3 
to  the  index  term  that  they  had  selected,  it  represented  perfect  agreement  in 
their  interpretation  of  the  superlativeness  of  the  adjective  or  adverb  modify¬ 
ing  the  index  term.  Therefore,  the  3,3  cell  was  given  an  a  priori  weight  of 
zero  in  computing  k  since  perfect  agreement  should  receive  no  penalty.  If 
one  indexer  had  assigned  a  numerical  weight  of  3  to  the  index  term  selected 


TABLE  41 

THE  WEIGHTING  ALGORITHM  USED  IN  COMPUTING  WEIGHTED  KAPPA 
FOR  ASSESSING  THE  LEVEL  OF  AGREEMENT  IN  ASSIGNING  NUMERICAL  WEIGHTS 
TO  THE  INDEX  TERMS  SELECTED  IN  THE  RELIABILITY  STUDY 
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and  the  other  indexer  had  assigned  a  numerical  weight  of  2,  they  only  dis¬ 
agreed  by  one  position  on  the  ordinal  scale,  and  therefore,  the  3,2  cells  were 
given  an  a  priori  weight  of  one  in  computing  k  ,  penalizing  this  mild  dis¬ 
agreement  only  slightly.  In  the  extreme  case, ^ if  one  indexer  had  assigned  a 
numerical  weight  of  3  to  the  index  term  selected  and  the  other  indexer  had 
assigned  a  numerical  weight  of  -2,  they  disagreed  by  five  positions  on  the 
ordinal  scale,  and  therefore,  the  3,-2  cells  were  given  an  a  priori  weight  of 
five  in  computing  k  ,  penalizing  this  extreme  disagreement  the  maximum  possible. 
This  same  logic  was^applied  in  determining  the  weights  to  be  used  in  computing 
k  throughout  the  remainder  of  the  matrix.  All  of  the  diagonal  cells  were  giv¬ 
en  a  weight  of  zero  since  in  no  case  should  perfect  agreement  be  penalized. 

All  cells  immediately  off  the  diagonal  were  penalized  by  a  weight  of  one; 
those  cells  slightly  farther  off  the  diagonal  were  penalized  by  a  weight  of 
two;  and  so  on  out  to  a  penalty  weight  of  five  for  the  case  of  worst  disagree¬ 
ment  . 


The  formula  for  computing 
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Table  42  shows  the  results  of  the  correlational  analysis  and  the  weighted 
kappa  analysis  for  the  six  pairwise  comparisons  between  the  four  reliability 
indexers  in  assigning  numerical  weights  to  each  index  term  selected,  based  on 
the  modifying  adjectives  and  adverbs,  for  the  entire  reliability  study  data 
base.  The  results  of  the  correlational  analysis  are  shown  first  in  Table  42. 
The  best  agreement  in  assigning  numerical  weights  to  each  index  term  selected 
was  obtained  between  the  experienced  indexer  and  inexperienced  indexer  A,  a 
correlation  coefficient  of  .80.  The  best  agreement  in  selecting  index  terms 
themselves  was  also  achieved  between  this  same  pair  of  indexers  (see  Table  40) . 


RESULTS  OF  THE  CORRELATIONAL  ANALYSIS  AND  THE  WEIGHTED  KAPPA  ANALYSIS 
FOR  THE  SIX  PAIRWISE  COMPARISONS  BETWEEN  THE  FOUR  RELIABILITY  INDEXERS 
IN  ASSIGNING  NUMERICAL  WEIGHTS  TO  EACH  INDEX  TERM  SELECTED 
FOR  THE  ENTIRE  RELIABILITY  STUDY  DATA  BASE 
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A  z  of  3.29  is  significant  at  the  .001  level  of  probability.  Therefore,  all  of  the  z  values  report¬ 
ed  in  this  table  are  highly  significant  and  lead  to  rejection  of  the  null  hypothesis  that  the  ob¬ 
tained  k  does  not  exceed  the  chance  level  of  agreement. 
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These  findings  corroborate  each  other  in  suggesting  that  an  individual  without 
a  research  background  in  only  six  training  sessions  can  be  taught  not  only  how 
to  select  the  most  appropriate  index  terms  but  also  how  to  consistently  assign 
weights  to  these  terms  based  on  the  modifying  adjectives  and  adverbs.  The 
other  correlations  reported  in  Table  42  are  lower,  but  none  is  less  than  .64. 

All  six  correlation  coefficients  are  significantly  different  from  zero  well 
beyond  the  .001  level  of  probability. 

On  the  weighted  kappa  side  of  Table  42  the  first  column  shows  the  value 
of  k  ;  the  second  column  shows  the  standard  error  of  k  ;  and  the  last  column 
lists  the  normal  deviate,  z,  obtained  by  dividing  k  by  its  standard  error. 

All  of  the  z  values  are  large,  and  consequently,  highly  significant,  indicat¬ 
ing  that  in  all  six  comparisons  the  null  hypothesis  that  the  obtained  k  does 
not  exceed  the  chance  level  of  agreement  can  be  rejected.  As  was  expected, 
the  k  values  are  similar  in  magnitude  to  their  correlation  coefficient  coun¬ 
terparts.  Again,  the  best  agreement  as  measured  by  weighted  kappa  was  ob¬ 
tained  between  the  experienced  indexer  and  inexperienced  indexer  A,  a  k  of 
.78.  The  three  values  shown  in  parentheses  after  the  first  three  k  fs 
listed  in  Table  42  were  computed  in  order  to  determine  the  level  of  agreement 
achieved  if  those  instances  were  excluded  where  the  experienced  indexer  did 
not  select  an  index  term,  and  consequently,  did  not  assign  a  numerical  weight 
but  the  other  less  experienced  indexer  did  select  an  index  term  and  assigned 
a  weight  to  it.  This  proved  to  be  the  area  of  major  confusion  in  executing 
the  reliability  study  as  pointed  out  earlier  in  this  section  in  discussing 
the  results  of  the  kappa  analysis  of  level  of  agreement  in  selecting  the  index 
terms  themselves.  Instances  where  the  experienced  indexer  did  not  assign  a 
weight  but  the  other  indexer  did  form  one  row  in  the  weighted  kappa  computa¬ 
tional  matrix.  This  row  can  be  omitted  from  the  computation,  resulting  in  a 
value  for  k  that  ignores  this  major  area  of  confusion  and  takes  into  account 
only  those  instances  where  both  indexers  selected  an  index  term,  and  conse¬ 
quently,  assigned  a  weight.  The  gain  in  the  value  of  k  is  not  very  large  for 
the  comparison  between  the  experienced  indexer  and  inexperienced  indexer  A 
when  k  was  recomputed  in  this  fashion.  However,  the  gain  was  quite  substan¬ 
tial  in  the  other  comparisons  between  the  experienced  indexer  and  the  princi¬ 
pal  investigator  and  between  the  experienced  indexer  and  inexperienced  indexer 
B.  This  suggests  that  with  additional  training  to  clarify  this  area  of  con¬ 
fusion  and  with  more  indexing  experience,  the  level  of  agreement  among  the 
four  reliability  indexers  could  possibly  be  raised  to  a  value  of  .80  to  .85 
as  measured  by  any  of  the  three  agreement  statistics  employed  in  this  study. 
However,  values  in  the  .90fs  are  the  ultimate  objective. 

In  conclusion,  it  might  be  of  interest  to  point  out  that  the  initial  ex¬ 
pectation  in  beginning  this  reliability  study  was  that  it  would  be  extremely 
difficult  to  train  nonresearch-oriented  individuals  to  consistently  index  the 
narrative  sections  of  Evaluation  Report  forms  using  the  complex  content  analy¬ 
sis  methodology  that  had  been  developed  in  the  pilot  study.  The  surprising 
result  is  that  in  only  six  training  sessions  a  quite  respectable  level  of  agree¬ 
ment  was  achieved.  Moreover,  one  of  the  inexperienced  indexers  showed  a  higher 
level  of  agreement  with  the  experienced  indexer  than  the  principal  investigator 
did,  and  the  other  inexperienced  indexer  agreed  with  the  experienced  indexer 
almost  as  well  as  the  principal  investigator.  The  intuitive  feeling  that  the 
reliability  indexers  had  after  completing  the  reliability  study  was  that  the 
most  difficult  part  of  learning  to  index  consistently  was  over  and  that  with 
additional  practice  and  some  review  training  sessions  they  could  improve  their 
indexing  skill. 
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SECTION  6.  FUTURE  AREAS  OF  INVESTIGATION 

In  the  subsequent  10-month  period  of  this  research  project  beginning 
March  1,  1973  and  concluding  December  31,  1973,  additional  studies  of  the  225 
Evaluation  Reports  in  the  pilot  study  sample  and  the  444  Evaluation  Reports  in 
the  cross  validation  and  generalization  samples  will  be  carried  out  in  order 
to  devise  valid,  short-cut  methods  of  indexing  the  narrative  content  of  these 
reports  based  on  the  more  lengthy  and  complex  content  analysis  methodology 
that  has  already  been  developed.  The  issues  of  trainability  and  reliability 
in  indexing  this  type  of  narrative  text  will  be  studied  further.  The  follow¬ 
ing  specific  tasks  are  being  undertaken: 

A.  Short-cut  Indexing  Methods 

Efforts  are  in  progress  to  develop  valid,  short-cut  methods  for  index¬ 
ing  the  narrative  content  of  Evaluation  Reports  by  capitalizing  on  the  findings 
resulting  from  the  various  stepwise  discriminant  analyses  that  have  been  per¬ 
formed.  The  variables  that  are  entered  into  the  discriminant  function  at  the 
first  five  to  ten  steps  in  the  analysis  appear  to  be  the  key  variables  in  dis¬ 
criminating  among  the  three  criterion  groups.  These  variables  then  form  one 
target  for  study,  i.e.,  how  to  extract  this  differentiating  information  from 
the  narrative  text  in  a  simple  but  reliable  fashion  that  will  achieve  as  good 
or  nearly  as  good  classification  accuracy  as  the  longer,  more  complex  index¬ 
ing  methodology.  This  approach  is  being  used  to  develop  optimum  streamlined 
classification  algorithms  for  all  four  ratings  represented  in  the  research 
data  base  studied  thus  far,  i.e.,  AT’s,  BT’s,  CSfs,  and  RM’s. 

In  addition,  the  literature  is  being  searched  for  work  that  may  be 
relevant  to  the  objective  of  developing  valid,  short-cut  methods  for  indexing 
the  narrative  content  of  Evaluation  Reports. 

B.  Extension  of  the  Inter-indexer  Reliability  Study 

An  extension  of  the  reliability  study  is  being  conducted  using  four 
reliability  indexers.  Two  of  the  original  reliability  indexers  (inexperienced 
indexers  A  and  B),  after  refresher  training  using  the  updated  version  of  the 
training  manual,  will  independently  index  a  different  set  of  48  Evaluation  Re¬ 
ports  than  that  used  in  the  original  reliability  study  and  their  indexing  de¬ 
cisions  will  be  compared  to  the  judgments  of  the  experienced  indexer.  The 
level  of  agreement  between  each  of  the  inexperienced  indexers  and  the  experi¬ 
enced  indexer  after  a  second  exposure  to  reliability  indexing  will  be  calcu¬ 
lated  to  determine  if  the  level  of  agreement  can  be  improved  with  additional 
training  and  experience.  In  addition,  two  new  reliability  indexers  will  be 
trained,  and  they  will  independently  index  the  original  set  of  48  Evaluation 
Reports.  Their  indexing  decisions  will  be  compared  to  those  of  the  experienc¬ 
ed  indexer  to  determine  if  the  level  of  agreement  achieved  is  comparable  to 
that  found  in  the  first  reliability  study. 
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PURPOSE 

The  purpose  of  this  manual  is  to  train  nonresearchers  in  the  content 
analysis  techniques  developed  in  a  personnel  research  study  to  analyze  the 
narrative  sections  of  Navy  performance  evaluations  for  senior  enlisted  person¬ 
nel  in  pay  grades  E-7  (Chief  Petty  Officer),  E-8  (Senior  Chief  Petty  Officer), 
and  E-9  (Master  Chief  Petty  Officer),1  The  objective  of  this  study  was  to 
provide  personnel  decision  makers  (e.g.,  selection  boards  and  detailers)  with 
a  standardized  way  of  detecting  valid  and  discriminating  indicators  of  on-job 
performance  in  narrative  evaluation  comments. 


BACKGROUND 

Section  19,  Evaluation  Section,  of  Evaluation  Report  Form  NAVPERS  1616/8* 
is  designed  to  permit  the  rater  (evaluator)  to  compare  the  ratee  with  all 
others  of  his  rate  known  to  the  rater  on  13  specific  aspects  of  on-job  perform¬ 
ance  as  well  as  to  make  an  overall  comparative  evaluation  of  the  ratee fs  per¬ 
formance  (see  Figure  A-l) .  Ratings  are  made  by  marking  the  column  of  the  rat¬ 
ing  distribution  into  which  the  rater  evaluates  that  the  ratee  falls  (for  ex¬ 
ample,  top  1%  for  superlative  performance).  Section  19R  of  this  form  provides 
space  for  the  rater  to  write  narrative  evaluation  comments  to  describe  further 
the  ratee* s  performance  and  qualifications.  Section  19S  of  this  form  provides 
space  for  the  rater  to  write  narrative  justification  comments  and  is  required 
to  support  any  marks  assigned  to  the  top  or  bottom  10,  5,  or  1%  columns  of 
Section  19. 

Sections  19R  and  19S  are  referred  to  as  the  narrative  text  of  the  Evalua¬ 
tion  Report  since  they  are  the  only  portions  of  the  report  where  the  rater 
uses  his  own  words  to  assess  the  on-job  performance  of  the  senior  enlisted  man 
that  he  is  rating.  Thus  far  the  narrative  evaluation  and  justification  sec¬ 
tions  of  the  Evaluation  Report  have  not  been  exploited  systematically  in  mak¬ 
ing  personnel  decisions  because  narrative  text  tends  to  resist  objective  analy¬ 
sis  and  interpretation.  However,  results  from  a  content  analysis  study  of  the 
narrative  text1  strongly  suggest  that  there  are  stable  differences  among  the 
performance  characteristics  of  chief  petty  officers  that  are  reflected  in  the 
narrative  statements  written  about  them  by  evaluators.  Furthermore,  these 
differences  are  both  identifiable  and  quantifiable.  The  remainder  of  this 
manual  presents  a  set  of  explicit  and  detailed  guidelines  for  identifying,  in¬ 
dexing  or  labeling,  and  quantifying  (by  means  of  a  weighting  scale)  the  con¬ 
cepts  and  ideas  represented  in  the  narrative  text  of  Navy  performance  evalua¬ 
tions  for  senior  enlisted  personnel.  These  quantified  labels  have  been  shown 
to  discriminate  or  differentiate  between  superior  ratees  and  their  slightly 
less  qualified  colleagues. 


* 


This  form  subsequently  has  been  replaced  by  another  form  that  can  be  scanned 
by  an  optical  character  reader;  however,  the  content  of  the  two  forms  is 
essentially  the  same. 
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RATEES  AS  MANAGERS 

Senior  enlisted  personnel  in  pay  grades  E-7/8/9  are  all  managers  in  the 
sense  that  they  all  are  responsible  for  the  supervision  of  other  enlisted  men 
whose  work  they  direct.  Therefore,  the  unifying  focus  in  this  manual  is  on 
the  assessment  of  a  chief  petty  officer  (ratee)  as  a  manager.  The  operations 
of  a  manager  may  differ  from  one  organization  or  from  one  institutional  set¬ 
ting  to  another;  however,  the  functions  of  a  manager  are  common  to  all.  The 
task  of  the  manager  is  one  of  selecting  goals  and  designing  and  maintaining 
an  environment  that  makes  possible  the  performance  of  individuals  working  to¬ 
gether  in  a  group  to  attain  these  goals.  Chief  petty  officers  in  pay  grades 
E-7/8/9  are  junior  level  managers,  and  as  such,  they  must  perform  technical  as 
well  as  managerial  functions.  Table  A-l  shows  a  hierarchy  of  29  index  terms 
or  labels  that  can  be  used  to  characterize  the  on- job  managerial  performance 
of  chief  petty  officers.  These  index  terms  are  the  terms  to  be  imposed  onto 
the  narrative  text  to  give  it  objective  structure  and  to  systematize  the  way 
that  this  text  is  analyzed  and  interpreted.  Note  in  Table  A-l  that  the  29 
index  terms  are  divided  into  three  sections.  The  first  section  contains  seven 
specific  MANAGEMENT  FUNCTIONS  that  many  authorities  on  management  practice 
agree  are  the  characteristic  duties  of  all  managers . 2  *  3 >4 * 5 Although  some 
authorities  believe  that  there  are  more,  less,  or  different  functions  perform¬ 
ed  by  managers,  these  seven  functions  were  selected  because  they  are  repre¬ 
sentative  of  the  duties  that  chief  petty  officers  actually  perform. 

The  second  section  of  Table  A-l  contains  index  terms  for  13  specific 
SKILLS  AND  ABILITIES  considered  to  be  important  by  Navy  supervisory  personnel 
in  performing  effectively  as  a  chief  petty  officer.  While  some  authorities  on 
management  practice  consider  making  a  judgment  about  whether  or  not  an  indi¬ 
vidual  possesses  a  skill,  quality,  or  ability  to  be  a  subjective  process,  Navy 
evaluators  do  repeatedly  call  out  these  specific  qualities  in  their  narrative 
evaluations  because  many  of  these  qualities  are  dimensions  on  which  they  rate 
the  ratee  in  Section  19  of  the  Evaluation  Report.  The  first  section  of  Table 

A-l - MANAGEMENT  FUNCTIONS - deals  with  how  a  ratee  performs  his  managerial 

functions  and  is  result  oriented,  while  the  second  section - SKILLS  AND  ABILI¬ 
TIES - contains  index  terms  that  relate  to  an  individual's  characteristics  and 

qualities  which,  if  used,  may  help  him  achieve  good  results. 

The  third  section  of  Table  A-l - PRODUCTIVITY  AND  ACHIEVEMENT - is  the 

most  result-oriented  section  of  the  hierarchy.  Here  are  included  the  measures 
of  overall  performance.  DRIVE  and  SERVICE  MOTIVATION  (a  specific  type  of 
drive)  are  included  in  this  section  since  drive  is  considered  to  contribute  to 
successful  performance.  POTENTIAL  also  is  included  here  since  potential  is  a 
measure  of  future  performance.  AWARDS  AND  PUNISHMENT,  REPUTE,  and  ASSET  TO 
THE  NAVY  represent  acknowledgments  of  an  individual's  performance,  either 
positive  or  negative  acknowledgment. 


QUANTIFYING  THE  INDEX  TERMS 

It  is  not  enough  to  simply  label  a  narrative  statement  with  the  most  ap¬ 
propriate  index  term  since  the  statement  may  have  been  a  highly  positive, 
quite  positive,  neutral,  quite  negative,  or  highly  negative  one.  For  example, 
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TABLE  A-l 


HIERARCHY  OF  INDEX  TERMS 


MANAGEMENT  FUNCTIONS 
CONTROLLING 

LEADERSHIP  AND  DIRECTING 

ORGANIZATION 

PLANNING 

REPRESENTATION 

STAFFING 

USE  OF  COMMUNICATION 


SKILLS  AND  ABILITIES 
COMMUNICATION 

CONDUCT,  INTEGRITY,  AND  PRIDE 

COOPERATION 

ENDURANCE 

FLEXIBILITY 

GROOMING  AND  ATTIRE 

INITIATIVE 

INTELLECTUAL  FUNCTIONING 

PROFESSIONALISM 

RELIABILITY  AND  DEPENDABILITY 

RESOURCEFULNESS 

RESPONSIVENESS 

TECHNICAL  SKILLS 


PRODUCTIVITY  AND  ACHIEVEMENT 
AWARDS  AND  PUNISHMENT 
DRIVE 

SERVICE  MOTIVATION 
POTENTIAL 
REPUTE 


ASSET  TO  THE  NAVY 
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in  order  to  differentiate  between  the  ratee  who  plans  superbly  and  the  ratee 
who  plans  inadequately,  a  weighting  scale  was  devised  to  be  applied  to  each 
index  term  that  is  used  (see  Table  A-2).  The  weighting  scale  contains  five 
numerical  values  ranging  from  5,  the  positive  end  of  the  scale,  to  1,  the 
negative  end  of  the  scale.  Under  each  numerical  value  in  Table  A-2  there  are 
listed  examples  of  adjectives  or  adverbs  that  may  be  used  by  the  rater  to  de¬ 
scribe  a  ratee* s  performance.  These  lists  of  words  provide  clues  to  the  in¬ 
dexer  as  to  which  numerical  value  to  assign  to  an  index  term.  As  a  simple  ex¬ 
ample,  if  the  rater  commented  that  the  ratee  was  highly  cooperative,  this 
statement  would  be  indexed  as  COOPERATION  and  assigned  a  weight  of  4  since 
highly  is  listed  as  an  example  under  numeral  4  in  Table  A-2. 

However,  in  many  cases  the  indexer  will  have  to  exercise  his  own  judgment. 
Some  of  the  words  that  will  require  such  a  judgment  are  the  following:  abnor¬ 
mally,  absolutely ,  all,  always ,  consistently ,  constantly ,  fully }  immeasurable , 
intense,  no  doubt,  obvious,  totally,  unquestionably,  and  uppermost .  An  index¬ 
er  will  encounter  many  more  words  than  those  enumerated  above  that  will  require 
good  judgment  in  choosing  the  most  appropriate  weight  to  use.  For  example, 
consider  the  following  statement:  **He  is  always  resourceful .  **  This  statement 
would  be  indexed  as  RESOURCEFULNESS  4,  because  the  statement  could  have  been 
worded  in  a  stronger  way.  The  rater  could  have  said,  "He  is  always  fully  re¬ 
sourceful."  This  stronger  statement  would  have  been  indexed  as  RESOURCEFUL¬ 
NESS  5.  The  words  always  and  fully  in  most  cases  indicate  the  use  of  a  4  on 
the  weighting  scale;  yet  in  the  above  example  where  they  appear  together,  the 
statement  is  made  so  strongly  that  a  weight  of  5  is  the  correct  indexing  de¬ 
cision.  However,  there  are  other  4-weighted  modifiers  that  when  used  in  com¬ 
bination  still  remain  a  4,  for  example,  remarkably  well  or  highly  effective , 
because  the  modifying  phrase  could  have  been  worded  even  more  strongly,  for 
example,  extremely  well  or  most  effective .  It  is  important  to  remember  when 
using  the  weighting  scale  that  the  indexer  should  ask  himself,  "Could  this 
statement  have  been  phrased  in  another  way  by  the  rater  that  would  have  made 
it  a  stronger  or  a  weaker  statement?**  The  indexing  convention  to  be  followed 
for  the  modifier  quite  is  to  consider  it  to  have  a  positive  connotation  and 
to  give  it  a  weight  of  4  as  in  the  following  example:  "He  has  been  quite 
resourceful  in  making  do  with  available  parts.**  RESOURCEFULNESS  4.  However, 
there  may  be  instances  in  which  the  indexer  would  assign  a  2  weight  to  the 
modifier  quite  if  the  context  was  sufficiently  negative  to  warrant  it  as  in 
the  following  example:  "His  tendency  to  be  quite  overweight  detracts  from  his 
overall  appearance. "  GROOMING  AND  ATTIRE  2.  Note  that  AWARDS  AND  PUNISHMENT 
is  given  either  a  5  or  a  1  weight  since  there  is  no  degree  of  variance.  Either 
the  ratee  was  given  an  award  or  not,  or  was  punished  (disciplined)  or  not. 


SPECIAL  INDEXING  CONSIDERATIONS 

An  alphabetical  dictionary  of  the  29  index  terms  appears  at  the  end  of 
this  discussion.  For  each  term  in  the  dictionary,  a  definition  is  given,  ex¬ 
amples  of  narrative  text  indexed  with  the  term  are  cited,  and  usage  rules  to 
guide  the  indexer  in  choosing  this  term  or  another  term  are  supplied.  Careful 
study  of  the  dictionary  will  instruct  the  new  indexer  in  how  index  terms  and 
their  numerical  weights  should  be  assigned  in  order  to  ensure  a  systematic  and 
objective  application  of  the  indexing  procedures  explained  in  this  manual. 
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TABLE  A- 2 
WEIGHTING  SCALE 


excellent 


4 

good 


3 

average 


2 

poor 


poorest 


superlative 


comparative 


comparative 


superlative 


best 

better  than 

average 

not  as  good 

worst 

most 

EXAMPLES 

as  most 

above 

above  average 

adequate 

declining 

bottom 

reproach 

better 

aptly 

quality 

least 

beyond 

commendable 

capable 

deficiency 

lowest 

reproach 

complete 

competent 

detrimental 

boundless 

deep 

generally 

fair 

exceptional 

definitely 

moderate 

in  need  of 

extra- 

easily 

satisf ac- 

insufficient 

ordinary 

effective 

tory 

lack  of 

extremely 

efficient 

sufficient- 

lower  than 

finest 

eloquent 

iy 

average 

flawless 

eminent 

usually 

lowering  of 

greatest 

exceeds 

negatively 

highest 

excels 

spotty 

ideal 

exemplary 

unfortunate 

little  to  be 

expeditious 

unwisely 

desired 

experienced 

weak  in 

limitless 

expertise 

with  the  ex¬ 

maximum 

extensive 

ception  of 

most 

favorable 

never 

great 

outstanding 

high/highly 

paramount 

immaculate 

perfect 

profound 

immensely 

impeccable 

4-good  (Cont.) 

sterling 

impressive 

rare 

superb 

innate 

remarkable 

superior 

inspires 

significantly 

surpassed  by 

instills 

skillful 

none 

invaluable 

smoothly 

top/topnotch 

keen 

solid 

unimpeachable 

laudable 

strongly 

unique 

leading 

surpassed 

unlimited 

marked 

thorough 

unmatched 

meticulously 

tremendous 

utmost 

model 

truly 

without  equal 

much 

unstinting 

without 

noteworthy 

valuable 

exception 

particularly 

vast 

100% 

rapidly 

very 

NOTE:  AWARDS 

AND  PUNISHMENT  is  assigned  a  weight 

of  either  5  or  1. 
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Although  some  indexing  examples  may  not  always  seem  logical  to  the  new  indexer, 
each  indexing  decision  has  been  meticulously  and  thoroughly  considered.  The 
examples  presented  in  the  alphabetic  dictionary  represent  a  distillation  of 
two  years1  of  indexing  experience  and  constitute  a  self-instructional  compila¬ 
tion  of  crucial  indexing  rules  and  conventions  that  the  new  indexer  needs  to 
know  in  depth  in  order  to  be  able  to  index  the  narrative  text  of  Evaluation 
Reports  accurately  and  consistently. 

There  are  several  indexing  considerations  that  should  be  kept  in  mind  as 
they  will  assist  the  indexer  in  maintaining  consistency  and  will  help  resolve 
indexing  dilemmas.  Sections  4A  and  4B  of  Evaluation  Report  Form  NAVPERS  1616/8 
provide  a  description  of  the  ratee’s  primary  and  collateral  duties.  These  sec¬ 
tions  should  be  read  as  background  information  before  beginning  to  index  the 
narrative  text.  An  example  of  these  two  sections  is  shown  below: 

4A.  DESCRIPTION  OF  PRIMARY  DUTIES  DURING  THIS  REPORTING  PERIOD 

Ratee  is  assigned  as  supervisor  of  Intermediate  Level  Mainte¬ 
nance  on  communications,  navigation,  and  radar  systems  in¬ 
stalled  in  the  EA-3B,  EP-3B,  and  EC-121M  aircraft. 

4B.  DESCRIPTION  OF  COLLATERAL  DUTIES  AND/OR  SPECIAL  ASSIGNMENTS 
DURING  THIS  REPORTING  PERIOD 

Ratee  is  assigned  on  a  rotational  basis  as  the  Avionics  Super¬ 
visor  at  the  squadron  detachment  in  Danang,  RVN.  He  also 
stands  Assistant  Squadron  Duty  Officer  watches. 

Information  about  a  ratee fs  job  duties  is  needed  when  indexing  because  a 
statement  about  an  individual’s  achievement  may  not  be  in  terms  of  performing 
a  managerial  function  but  rather  in  terms  of  his  overall  performance.  Consid¬ 
er  the  following  statement  from  the  narrative  text:  ’’Ratee’s  supervision  of 
maintenance  of  the  communication  and  radar  systems  has  been  outstanding.” 

This  statement  would  be  indexed  as  PRODUCTIVITY  AND  ACHIEVEMENT  5  because  it 
is  an  assessment  of  the  ratee’s  overall  job  performance  as  a  supervisor  of 
intermediate-level  maintenance  rather  than  performance  of  the  controlling  func¬ 
tion  of  maintaining  equipment  in  order  to  assure  accomplishment  of  plans. 

Another  consideration  to  keep  in  mind  is  that  when  a  description  of  a  job 
or  job  duties  is  included  in  the  narrative  text,  this  description  is  not  in¬ 
dexed  since  it  is  a  factual  statement  describing  the  qualifications  needed  to 
perform  a  specific  job  or  the  duties  of  that  job.  Therefore,  the  statement  is 
about  the  job  itself  and  not  about  the  ratee.  Even  if  such  a  statement  is 
modified  by  adjectives  or  adverbs,  it  still  is  not  indexed  if  it  refers  to  how 
a  job  should  be  performed  and  not  to  how  the  ratee  actually  performs  a  job. 

As  an  illustration  of  this  convention,  if  planning  is  mentioned  as  one  of  the 
ratee’s  duties,  this  is  a  factual  statement  since  it  is  the  duty  of  all  manag¬ 
ers  to  plan  and  no  index  term  would  be  assigned  to  this  statement.  For  example, 
’’Chief  XX  is  required  to  develop  procedural  methods  of  accomplishing  the  divi¬ 
sion  workload.”  Even  if  an  adjective  or  adverb  is  added  to  this  statement - 

"Chief  XX  is  required  to  develop  effective  and  efficient  procedural  methods  of 
accomplishing  the  division  workload,”  the  statement  is  still  about  a  specific 
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job  duty  or  requirement  and  the  modifiers  refer  to  how  the  job  should  be  per¬ 
formed.  The  modifiers  do  not  refer  to  the  ratee  per  se  and,  therefore,  this 
statement  should  not  be  indexed  since  it  is  not  evaluating  the  ratee.  However, 
if  a  statement  is  a  qualitative  statement  and  refers  to  the  ratee,  then  it  be¬ 
comes  a  statement  of  evaluation  and  is  indexed.  If  the  rater  said  that  the 
ratee  plans  well  on  the  job,  then  a  value  judgment  has  been  rendered  about  what 
kind  of  a  planner  the  ratee  is.  For  example,  "Chief  XX  has  developed  effective 
and  efficient  procedural  methods  of  accomplishing  the  division  workload";  this 
statement  would  be  labeled  PLANNING  4.  The  statement  no  longer  is  a  factual  one 
but  has  become  an  evaluative  comment  about  the  ratee  rather  than  a  statement  de¬ 
scribing  a  job  requirement.  "Chief  XX  has  developed  procedural  methods  of  accom¬ 
plishing  the  division  workload,"  would  be  labeled  PLANNING  3  since  the  rater 
thought  it  worthwhile  to  mention  this  information  and  the  statement,  therefore, 
evaluates  the  ratee  although  no  modifying  adjective  or  adverb  was  used.  Always 
keep  in  mind  that  indexing  decisions  are  made  in  terms  of  the  qualifications 
that  a  ratee  actually  possesses  that  can  aid  him  in  performing  a  job ,  in  terms 
of  how  a  job  is  performed  by  a  ratee,  or  in  terms  of  the  results  achieved . 

Also  keep  in  mind  that  in  deciding  on  which  numerical  weight  to  use,  modi¬ 
fying  adjectives  and/or  adverbs  must  be  associated  with  the  idea  or  concept  be¬ 
ing  indexed  and  not  with  another  idea  or  concept  in  the  same  sentence.  For 
example,  consider  the  following  statement:  "His  resourcefulness  in  completing 
his  tasks  in  the  most  efficient  and  thorough  manner  is  noted."  Most  is  associ¬ 
ated  with  the  manner  in  which  the  ratee  performs  his  tasks  and,  therefore,  quali¬ 
fies  PRODUCTIVITY  AND  ACHIEVEMENT  as  a  5.  The  rateefs  resourcefulness  is  not 
modified  but  it  is  stated  that  he  possesses  that  characteristic.  It  helped  to 
make  the  5  weight  possible  for  PRODUCTIVITY  AND  ACHIEVEMENT,  but  the  first  part 
of  the  statement  is  only  indexed  as  RESOURCEFULNESS  3.  To  be  given  a  5  weight, 
the  statement  would  have  had  to  have  been,  "His  outstanding  resourcefulness  in 
completing  his  tasks  in  the  most  efficient  and  thorough  manner  is  noted."  When 
there  is  no  modifier  given  for  an  evaluative  statement,  a  3  weight  is  assigned 
to  the  index  term  selected  (e.g. ,  "His  planning  efforts  have  led  to..."  would 
be  labeled  PLANNING  3).  Also,  if  a  modifying  adjective  or  adverb  that  falls 
at  the  3  position  on  the  weighting  scale  is  included  in  the  evaluative  state¬ 
ment,  the  index  term  selected  still  would  be  given  a  weight  of  3  (e.g.,  "His 
competent  planning  has  led  to..."  would  also  be  labeled  PLANNING  3).  Each 
statement  indexed  has  to  be  regarded  as  a  separate  entity  lest  confusion  and 
inconsistency  result.  Always  remember  the  rule:  How  else  could  this  statement 
have  been  worded? 

However,  if  an  adjective  occurs  before  a  string  of  words  and  phrases, 
this  adjective  modifies  each  word  or  phrase  in  the  string  until  there  is  a 
clear  break  in  the  sentence  structure,  or  until  the  adjective  could  not  logi¬ 
cally  and/or  grammatically  be  associated  with  a  particular  word  or  phrase. 

For  example,  "His  outstanding  technical  knowledge  and  organizational  ability 
have  contributed  to..."  would  be  indexed  as  TECHNICAL  SKILLS  5  and  ORGANIZA¬ 
TION  5.  The  adjective  outstanding  modifies  both  phrases.  Note  that  organiza¬ 
tion  in  the  above  example  is  referred  to  as  a  skill;  yet  it  is  placed  under 
MANAGEMENT  FUNCTIONS  in  the  hierarchy  of  index  terms.  It  is  often  difficult 
to  differentiate  between  the  performance  of  a  function  and  the  function  as  an 
actual  skill  that  an  individual  may  have.  For  example,  there  is  a  definite 
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ability  to  lead  or  skill  of  leadership;  yet  it  also  is  a  very  important  func¬ 
tion  performed  by  managers.  These  subtleties  in  word  meaning  and  usage  are 
part  of  the  expressive  fabric  of  the  English  language  and  continue  to  plague 
those  who  strive  to  achieve  precision  in  systematizing  the  information  content 
of  written  discourse.  At  some  point  arbitrary  rules  have  to  be  imposed.  In 
this  content  analysis  scheme,  both  organization  and  leadership  are  considered 
to  be  principally  management  functions  rather  than  skills  an  individual  may 
possess. 

It  is  also  important  to  remember  when  indexing  and  applying  the  weighting 
scale  to  be  careful  to  take  note  of  the  words  that  are  actually  contained  in 
the  definition  of  an  index  term.  Consider  the  definition  of  COMMUNICATION: 
COMMUNICATION  refers  to  the  expression  of  thoughts  and  feelings  through  the 
spoken  or  written  word  in  a  clear  and  concise  manner.  Therefore,  the  indexer 
should  regard  the  ability  to  communicate  in  a  clear  And  concise  manner  as  an 
average  ability  and  index  any  statement  phrased  similarly  as  COMMUNICATION  3. 

If  a  ratee  is  said  to  communicate  in  a  very  concise  manner,  then  this  state¬ 
ment  would  be  indexed  as  COMMUNICATION  4. 

Every  attempt  has  been  made  to  present  the  information  contained  in  this 
manual  in  as  explicit  and  lucid  a  form  as  possible.  However,  indexing  remains 
more  of  an  art  than  a  science  for  all  of  the  reasons  alluded  to  previously. 

As  an  indexer  you  will  encounter  segments  of  narrative  text  for  which  only 
your  considered  judgment  can  help  you  arrive  at  the  final  decision.  It  is  im¬ 
portant,  though,  that  you  try  to  keep  your  judgments  as  consistent  as  possible. 
The  best  way  to  assure  consistency  is  to  keep  records  of  difficult  or  marginal 
decisions  and,  if  possible,  of  the  basis  on  which  these  decisions  were  made. 
Table  A-3  presents  a  glossary  of  indexing  decisions  that  were  made  by  one  ex¬ 
perienced  indexer  to  handle  the  appearance  of  ambiguous  or  troublesome  words 
and  phrases  in  narrative  text.  Use  this  table  as  an  extra  indexing  guide. 

It  is  recommended  that  the  new  indexer  become  thoroughly  familiar  with 
this  training  manual  before  attempting  to  index  the  narrative  sections  of 
Evaluation  Reports.  If  he  can  compare  his  independent  trial  indexing  deci¬ 
sions  with  those  of  an  experienced  indexer,  this  procedure  will  serve  to  pin¬ 
point  areas  of  confusion  in  his  understanding  of  the  indexing  rules  and  con¬ 
ventions.  Frequent  rereading  of  the  manual  will  help  to  guarantee  that  the 
rules  are  applied  the  same  way  from  day  to  day. 
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TABLE  A-3 


GLOSSARY  OF  WORD  CLUES  TO  THE  USE  OF  INDEX  TERMS 


Acts  with  ease  -  CONDUCT,  INTEGRITY,  AND  PRIDE 
Aggressive  *  DRIVE 
Agreeable  ■  COOPERATION 

Alert  =  INTELLECTUAL  FUNCTIONING  (See  Also  RESPONSIVENESS) 

Amiable/Affable  =  COOPERATION 

Appraisal  (assessment)  of  personnel  =  STAFFING 

Background  in  rate  ■  TECHNICAL  SKILLS 

Bearing  -  CONDUCT,  INTEGRITY,  AND  PRIDE 

Can-do  attitude  ■  DRIVE 

Common  sense  -  INTELLECTUAL  FUNCTIONING 

Congenial  =  COOPERATION 

Constant  -  RELIABILITY  AND  DEPENDABILITY 

Coordinate  =  ORGANIZATION 

Decisiveness  =  SKILLS  AND  ABILITIES* 

Delegate  -  ORGANIZATION 
Deliberate  -  SKILLS  AND  ABILITIES* 

Demeanor  =  CONDUCT,  INTEGRITY,  AND  PRIDE 
Deportment  -  CONDUCT,  INTEGRITY,  AND  PRIDE 

Devoted  (dedicated)  to  duty  or  the  Navy  =  SERVICE  MOTIVATION 
Devotes  (dedicates)  long  hours  =  DRIVE 
Diligent  -  DRIVE 

Disposition  =  CONDUCT,  INTEGRITY,  AND  PRIDE 

Dynamic  =  DRIVE 

(his)  endeavor  =  DRIVE 

Expediently  =  RESPONSIVENESS 

Expeditious  =  RESPONSIVENESS 

Expertise  -  TECHNICAL  SKILLS 

Forceful  -  DRIVE 

Forcefulness  of  expression  =  COMMUNICATION 

Foresight  -  SKILLS  AND  ABILITIES 

Friendly  =  CONDUCT,  INTEGRITY,  AND  PRIDE 

Genial  -  CONDUCT,  INTEGRITY,  AND  PRIDE 

Humor  =  CONDUCT,  INTEGRITY,  AND  PRIDE 

Imagination  =  RESOURCEFULNESS 

Ingenuity  =  RESOURCEFULNESS 

Innovative  =  RESOURCEFULNESS 

(has)  instituted  =  INITIATIVE 

Inventory  =  PLANNING  +  CONTROLLING 

Judgment  =  INTELLECTUAL  FUNCTIONING 

Logical  (keen)  mind  =  INTELLECTUAL  FUNCTIONING 

Mature  =  RELIABILITY  AND  DEPENDABILITY 

New  ideas  =  INITIATIVE 


If  ratee  is  decisive  (deliberate)  in  his  thinking,  index 
as  SKILLS  AND  ABILITIES.  If  ratee  is  decisive  (deliberate) 
in  his  behavior,  index  as  CONDUCT,  INTEGRITY,  AND  PRIDE. 


TABLE  A- 3  (CONT.) 

GLOSSARY  OF  WORD  CLUES  TO  THE  USE  OF  INDEX  TERMS 


Original  thinking  =  INITIATIVE 

Perseverance  =  ENDURANCE 

Recommended  changes  ■  INITIATIVE 

Sincere  -  CONDUCT,  INTEGRITY,  AND  PRIDE 

Stable  =  RELIABILITY  AND  DEPENDABILITY 

Suggestion  making  *  INITIATIVE 

Supervising  men  =  LEADERSHIP  AND  DIRECTING 

Supervising  tasks  «  MANAGEMENT  FUNCTIONS 

Tactful  (used  alone)  »  CONDUCT,  INTEGRITY,  AND  PRIDE 

Tactful  with  men  or  subordinates  ■  LEADERSHIP  AND  DIRECTING 

Talent  =  SKILLS  AND  ABILITIES 

Traffic  flow  pattern  =  PLANNING  +  CONTROLLING 

Versatile  =  SKILLS  AND  ABILITIES 

Vigor  =  DRIVE 

Zeal  =  DRIVE 
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ALPHABETICAL  DICTIONARY  OF  INDEX  TERMS 


Format:  The  name  of  each  index  term  is  shown 
in  all  capital  letters  at  the  beginning  of  the 
definition  of  the  term.  The  definition  is  followed 
by  a  SEE  ALSO  entry  if  there  is  one.  The  last  sec¬ 
tion  of  the  format  presents  a  number  of  examples  of 
how  each  term  in  the  dictionary  was  used.  The 
number  following  each  index  term  is  the  weight 
assigned  to  it  by  the  indexer.  Additional  indexing 
clues  may  also  be  given. 


110 


ASSET  TO  THE  NAVY  refers  to  the  worth  or  value  of  having  an  individual  contin¬ 
ue  his  career  in  the  Navy. 

EXAMPLES: 


Ratee  has  been  an  exceptionally  ASSET  TO  THE  NAVY  5 

fine  asset  to  this  command. 


He  has  been  a  great  asset  to  ASSET  TO  THE  NAVY  A 

DESCOL  and  the  Navy. 


Ratee  has  proven  to  be  a  definite  ASSET  TO  THE  NAVY  A 

asset  with  his  outstanding  pro-  PROFESSIONALISM  5 

fessional  and  instructional  ability  COMMUNICATION  5 
in  a  relatively  short  time. 


He  is  an  asset  to  the  ship  and 
the  naval  service. 

Additional  Indexing  Clues: 

One  of  the  most  outstanding  men  in 
the  Navy  today. 

He  is  an  outstanding  military  man. 

Valuable  asset  to  the  service 

Great  value  to  the  U.S.  Navy 

Asset  to  this  command 

Valuable  to  any  command 

Credit  to  the  Navy 

Credit  to  the  squadron 

Asset  to  the  command  of  the  Navy 


ASSET  TO  THE  NAVY  3 


ASSET  TO  THE  NAVY  5 

ASSET  TO  THE  NAVY  5 
ASSET  TO  THE  NAVY  A 
ASSET  TO  THE  NAVY  A 
ASSET  TO  THE  NAVY  3 
ASSET  TO  THE  NAVY  3 
ASSET  TO  THE  NAVY  3 
ASSET  TO  THE  NAVY  3 
ASSET  TO  THE  NAVY  3 


ASSET  TO  THE  NAVY  is  to  be  used  only  when  the  individual  himself  is  con¬ 
sidered  to  be  an  asset  and  not  one  of  his  traits.  If  one  of  his  traits 
is  an  asset  to  the  Navy,  use  asset  as  a  qualifier  for  the  trait.  How¬ 
ever,  if  his  trait  is  an  asset  in  performing  a  specific  function,  asset 
then  qualifies  the  index  term  assigned  to  the  function.  For  example, 
"His  cooperative  nature  is  a  great  asset  to  the  Navy."  COOPERATION  A. 
However,  "His  cooperative  nature  is  a  great  asset  in  the  performance  of 
his  coordinating  responsibilities  with  Facilities  Control  and  the  Re¬ 
ceiver  Site,"  would  be  indexed  as  COOPERATION  3;  ORGANIZATION  A. 
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AWARDS  AND  PUNISHMENT  are  measurements  of  an  individual's  job  performance  as 
reflected  in  his  receiving  official  commendatory  recognition  or  penalty, 
or  his  being  recommended  for  such  an  award  or  penalty. 

EXAMPLES: 


BTC  was  recommended  for  an  achieve-  AWARDS  AND  PUNISHMENT  5 
ment  medal  for  his  leadership 
efforts  during  the  1969  WESTPAC 
Cruise. 


Awarded  the  Navy  Commendation 
Medal  for  meritorious  service 
while  attached  to  Fighter  Squadron 
ONE  FIVE  ONE  embarked  in  USS  CORAL 
SEA  (CVA-43)  during  combat  opera¬ 
tions  from  10  October  1968  to  30 
March  1969. 

He  was  awarded  an  oral  admonition. 
Additional  Indexing  Clues: 

Reported  in  a  commendatory  way 
Relieved  of  duties 


AWARDS  AND  PUNISHMENT  5 


AWARDS  AND  PUNISHMENT  1 


AWARDS  AND  PUNISHMENT  5 
AWARDS  AND  PUNISHMENT  1 


Use  AWARDS  AND  PUNISHMENT  only  once  no  matter  how  many  awards  a  ratee 
received  if  they  are  all  mentioned  in  one  statement.  For  example, 

"He  won  five  awards  for  outstanding  performances  last  year,"  would  be 
indexed  as  AWARDS  AND  PUNISHMENT  5. 

Use  this  term  even  if  the  award  is  given  to  a  ratee' s  squadron,  command, 
or  other  such  unit.  For  example,  "He  contributed  directly  to  the  annual 
OP-EVAL  award  of  'Outstanding'  given  this  division  by  the  DCA  Inspection 
Team  during  their  recent  visit  to  this  Command."  AWARDS  AND  PUNISHMENT  5. 

Do  not  index  that  part  of  the  sentence  following  the  award  statement 
which  tells  about  the  performance  for  which  the  ratee  won  the  award. 

This  is  all  considered  to  be  part  of  the  award  statement.  For  example, 
"Ratee  was  recommended  for  and  received  the  Navy  Achievement  Medal  for 
his  outstanding  professional  performance  during  this  reporting  period," 
would  be  indexed  simply  as  AWARDS  AND  PUNISHMENT  5. 


NOTE:  AWARDS  AND  PUNISHMENT  is  assigned  a  weight  of  either  5  or  1. 
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COMMUNICATION  refers  to  the  expression  of  thoughts  and  feelings  through  the 
spoken  or  written  word  in  a  clear  and  concise  manner.  Includes  public 
speaking ,  reports ,  records ,  and  letters . 

The  ability  to  communicate  is  reflected  in  the  choice  of  words  or  vocabu 
lary,  the  organization  of  the  expressed  thoughts,  phrasing,  sentence 
structure,  paragraphing,  and  overall  clarity  and  forcefulness  of  expres¬ 
sion. 

SEE  ALSO:  USE  OF  COMMUNICATION 
EXAMPLES: 


His  use  of  the  English  language  is 
excellent  and  he  expresses  himself 
exceptionally  well,  both  orally  and 
in  writing. 

COMMUNICATION  5 

COMMUNICATION  5 

Ratee's  ability  to  correctly  speak 
the  English  language  is  outstanding. 

COMMUNICATION  5 

He  utilizes  a  well-rounded  vocabulary 
to  very  effectively  express  himself. 

COMMUNICATION  4 

He  possesses  an  average  command  of 
the  English  language  both  orally 
and  in  writing. 

COMMUNICATION  3 

He  is  capable  of  expressing  himself 
clearly  and  adequately. 

COMMUNICATION  3 

He  is  soft  spoken  but  speaks  well; 
however,  his  written  work,  particu¬ 
larly  spelling,  could  use  improve¬ 
ment. 

COMMUNICATION  4 

COMMUNICATION  2 

Additional  Indexing  Clues: 


Excellent  knowledge  of  English 
language 

COMMUNICATION  5 

Command  of  language  superb 

COMMUNICATION  5 

Grammar  excellent 

COMMUNICATION  5 

Vocabulary  excellent 

COMMUNICATION  5 

His  reports  are  accurate. 

COMMUNICATION  4 

Expresses  well  in  writing 

COMMUNICATION  4 

Expresses  well  orally 

COMMUNICATION  4 

Expresses  well  orally  and  in  writing 

COMMUNICATION  4 

Speaks  with  ease 

COMMUNICATION  4 
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COMMUNICATION  (Continued) 


Can  converse  easily 

Eloquent  speaker 

Relaxed  group  speaker 

Is  at  ease  when  speaking 

Presents  matter  in  a  comprehensive 
and  interesting  manner 

Speaks  correctly 

Speaks  logically 

Verbal  expression  is  compre¬ 
hensive 

Capable  in  expressing  himself 
Difficulty  speaking  to  large  groups 


COMMUNICATION  4 


COMMUNICATION  4 


COMMUNICATION  3 


COMMUNICATION  3 


COMMUNICATION  3 


COMMUNICATION  3 


COMMUNICATION  3 
COMMUNICATION  3 


COMMUNICATION  3 
COMMUNICATION  2 


A  ratee's  ability  to  communicate  may  be  expressed  by  the  evaluator  with 
one  or  more  descriptive  phrases,  but  this  part  of  the  sentence  should  be 
treated  as  a  single  entity  and  labeled  only  once  with  COMMUNICATION. 

For  example,  "He  utilizes  a  well-rounded  vocabulary  to  very  effectively 
express  himself."  COMMUNICATION  4. 

Instructor  would  be  indexed  with  COMMUNICATION  unless  it  was  the  ratee's 
job  title  or  main  job  duty.  In  this  case  it  most  likely  would  be  labeled 
MANAGEMENT  FUNCTIONS  or  PRODUCTIVITY  AND  ACHIEVEMENT,  depending  upon  the 
statement. 
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CONDUCT,  INTEGRITY,  AND  PRIDE  refers  to  the  way  that  one  acts  and  to  the  men¬ 
tal  activities  and  attitudes  that  influence  behavior,  such  as  integrity 
and  pride.  Integrity  refers  to  one’s  moral  principles,  honesty,  and 
sincerity.  Pride  is  the  sense  of  one's  own  dignity  or  worth. 

For  the  assessment  of  integrity  we  would  look  at  an  individual's  use  of 
time  and  expense  reports,  dealings  with  others,  probity  in  handling 
assignments,  sincerity  about  his  work,  and  his  attitude  toward  life. 

EXAMPLES: 


His  conduct  and  personal  appearance 
are  always  superb. 

He  maintains  an  exemplary  military 
appearance,  and  has  high  moral 
standards  and  sound  sense  of  values. 

Ratee  reflects  pride  in  his  position 
as  a  Chief  Petty  Officer  and  pro¬ 
vides  an  outstanding  example  through 
his  exceptionally  fine  personal 
habits  and  dress  under  all  circum¬ 
stances. 

His  even  disposition  has  proven  an 
asset  in  maintaining  an  open  channel 
of  communication  between  instructor 
and  student. 

The  ratee  has  the  potential  to  be¬ 
come  an  outstanding  chief  but  has 
problems  controlling  his  drinking 
while  on  the  bench,  resulting  in 
tardiness  at  the  expiration  of 
liberty  and  his  absence  during 
working  hours. 

Excesses  in  alcohol  have  led  to 
tardiness  and  a  question  of  his 
dependability. 

Additional  Indexing  Clues: 

He  provides  an  excellent  example 
for  his  men. 


CONDUCT,  INTEGRITY,  AND  PRIDE  5 
GROOMING  AND  ATTIRE  5 

GROOMING  AND  ATTIRE  4 
CONDUCT,  INTEGRITY,  AND  PRIDE  4 
CONDUCT,  INTEGRITY,  AND  PRIDE  4 

CONDUCT,  INTEGRITY,  AND  PRIDE  3 
CONDUCT,  INTEGRITY,  AND  PRIDE  5 
GROOMING  AND  ATTIRE  5 


CONDUCT,  INTEGRITY,  AND  PRIDE  3 
USE  OF  COMMUNICATION  3 


POTENTIAL  5 
CONDUCT,  INTEGRITY, 
CONDUCT,  INTEGRITY, 
CONDUCT,  INTEGRITY, 


AND  PRIDE  2 
AND  PRIDE  2 
AND  PRIDE  2 


CONDUCT,  INTEGRITY,  AND  PRIDE  1 
CONDUCT,  INTEGRITY,  AND  PRIDE  2 
RELIABILITY  AND  DEPENDABILITY  2 


CONDUCT,  INTEGRITY,  AND  PRIDE  5 


Chief  XX' s  conduct  is  never  CONDUCT,  INTEGRITY,  AND  PRIDE  5 

questioned. 

Exemplary  behavior  CONDUCT,  INTEGRITY,  AND  PRIDE  4 
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CONDUCT,  INTEGRITY,  AND  PRIDE  (Continued) 


Professional  behavior 

CONDUCT,  INTEGRITY, 
PROFESSIONALISM  3 

AND 

PRIDE 

He  is  always  correct  and  proper 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

in  all  relationships. 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

Quick  humor 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

Gentlemanly 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

At  ease  with  superiors  and  sub¬ 
ordinates 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

Tactful 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

Friendly 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

Not  afraid  to  offer  criticism 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

Bordered  on  insubordination 

CONDUCT, 

INTEGRITY, 

AND 

PRIDE 

It  is  important  to  remember  that 

no  matter  how  positive  a  trait 

may  be 

it  is  impossible  to  be  consistent  in  weighting  these  traits;  therefore, 
the  weight  is  determined  by  the  adjective  which  further  qualifies  the 
type  of  trait  that  a  ratee  possesses.  For  example,  cheerful  or  cheerful 
personality  would  be  given  a  weight  of  3,  while  very  cheerful  or  a  very 
cheerful  personality  would  be  given  a  weight  of  4. 


Also  included  in  this  term  are  personality  traits  and  attitudes.  The 
following  in  their  present  form  would  all  be  indexed  as  CONDUCT,  INTEGRITY 
AND  PRIDE  3:  optimistic  attitude;  pleasant  attitude;  takes  pride  in  him¬ 
self;  pride  in  his  performance;  dignified;  self-confident;  upright;  honest 
sincere;  does  not  procrastinate;  spends  not  excessive  time  "visiting"; 
perfectionist;  quiet;  unselfish;  firm;  courage;  composed;  calm;  courtesy; 
disposition;  obedience;  loyalty  to  his  superiors;  friendly. 


Good  humor  and  a  good  example  for  contemporaries  both  would  be  indexed  as 
CONDUCT,  INTEGRITY,  AND  PRIDE  4  because  of  the  word  good. 


The  word  pride  often  appears  in  the  narrative  text  in  conjunction  with 
the  way  that  a  ratee  feels  about  the  Navy.  Under  these  circumstances, 
use  the  index  term  SERVICE  MOTIVATION. 
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CONTROLLING  is  the  measurement  of  performance  against  established  standards, 
correcting  deviations,  and  assuring  accomplishment  of  plans.  In  simple 
terms,  controlling  makes  certain  that  what  is  done  is  what  is  intended. 
Forward-looking  control  prevents  deviations  from  occurring  by  anticipat¬ 
ing  that  they  will  occur  unless  action  is  taken  now,  such  as  maintenance 
of  equipment. 

Any  activity  which  involves  feedback  is  a  controlling  function  in  that 
it  helps  to  ensure  the  successful  accomplishment  of  goals  (e.g.,  checks 
and  balances,  bookkeeping,  accounting  systems,  traffic  flow,  and  inven- 
tory). 

EXAMPLES: 


His  ability  to  seek  out  potential 
problems  and  correct  them  before 
failure  occurred  has  been  extremely 
beneficial. 

He  is  consistently  capable  of 
resolving  problem  areas  before  a 
critical  situation  can  develop. 

In  his  capacity  as  water  chief 
tender  he  has  enforced  a  strict 
and  professional  water  chemistry 
program. 

His  analysis  of  divisional  prob¬ 
lems,  both  functional  and  adminis¬ 
trative,  and  the  execution  of 
corrective  measures  have  been  very 
conducive  to  a  smooth  and  highly 
effective  division. 

Because  he  never  examined  the 
fireroom  equipment  during  the  10 
days  in  port,  the  unsatisfactory 
condition  of  the  boilers  went 
unchecked. 

Additional  Indexing  Clues: 

Prevents  problems 
Corrects  difficulties 
Overcomes  obstacles 


CONTROLLING  5 

CONTROLLING  4 

CONTROLLING  4 

PLANNING  3 
CONTROLLING  3 

PRODUCTIVITY  AND  ACHIEVEMENT  4 

CONTROLLING  1 


CONTROLLING  3 
CONTROLLING  3 
CONTROLLING  3 


Maintenance  is  a  controlling  function  because  it  helps  to  assure  accom¬ 
plishment  of  plans.  However,  if  an  individual’s  job  title  or  main  job 
duty  is  maintenance,  then  this  usually  would  be  indexed  as  PRODUCTIVITY 
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CONTROLLING  (Continued) 

AND  ACHIEVEMENT  since  maintenance  is  the  performance  of  his  total  job 
function. 

If  the  correction,  problem  solving,  or  prevention  is  of  a  technical  nature 
and  is  performed  solely  by  the  ratee  instead  of  by  a  group  of  his  subordi¬ 
nates,  it  should  be  regarded  more  as  a  technical  function  or  the  demon¬ 
stration  of  a  technical  skill  rather  than  considered  to  be  a  controlling 
function,  and  would  be  labeled  TECHNICAL  SKILLS.  For  example,  "Ratee' s 
knowledge  of  the  P-3  aircraft  electronic  systems  and  his  ability  to  expe¬ 
ditiously  correct  the  most  complex  electronic  problem  is  outstanding." 
TECHNICAL  SKILLS  5;  RESPONSIVENESS  3;  TECHNICAL  SKILLS  5.  The  modifying 
adverb  outstanding  belongs  to  both  phrases  of  this  sentence. 

The  words  assure  and  ensure  should  serve  as  a  clue  that  the  phrase  which 
follows  probably  will  be  indexed  as  a  controlling  function. 
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COOPERATION  is  acting  or  working  together  with  others  or  another  for  a  common 
purpose.  Includes  congenial }  amiable ,  or  agreeable  attitudes  which  en¬ 
hance  the  act  of  working  together. 

EXAMPLES: 


He  always  cooperates  fully  with  his 
seniors  and  accomplishes  his  duties 
in  an  outstanding  manner* 

He  is  always  willing  to  help  others 
in  any  way  he  can. 

Ratee  exemplifies  the  perfect 
officer-chief  relationship. 

Ratee  is  very  cooperative  with  his 
seniors  and  his  contemporaries, 
always  putting  the  interests  of 
the  Navy  first. 

He  is  cooperative  and  well  liked. 

He  is  a  pleasure  to  work  with. 

He  will  nevertheless  listen  to  the 
ideas,  beliefs,  and  suggestions  of 
others. 


COOPERATION  5 

PRODUCTIVITY  AND  ACHIEVEMENT  5 
COOPERATION  5 

COOPERATION  5 

COOPERATION  4 
SERVICE  MOTIVATION  5 

COOPERATION  3 
REPUTE  4 

COOPERATION  3 

COOPERATION  3 


Additional  Indexing  Clues: 


Never  hesitates  to  fully  cooperate 

Works  well  with  superiors  and 
subordinates 

Gets  along  well  with  others 

Continuous  cooperation  in  all  aspects 

Complete  cooperation 

Congenial 

Amiable 

Cooperative 

He  will  compromise. 

Cooperative  with  superiors  and 
subordinates 

Assists  others 

Uncooperative 


COOPERATION  5 
COOPERATION  4 

COOPERATION  4 
COOPERATION  4 
COOPERATION  4 
COOPERATION  3 
COOPERATION  3 
COOPERATION  3 
COOPERATION  3 
COOPERATION  3 

COOPERATION  3 
COOPERATION  2 


Usually  a  ratee1 s  interpersonal  relationship  with  his  subordinates  would 
be  indexed  as  LEADERSHIP  AND  DIRECTING.  However,  when  a  statement  has 
to  do  with  a  ratee’ s  cooperative  attitude  in  working  with  others,  whether 
superiors  or  subordinates,  then  it  should  be  labeled  COOPERATION. 
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DRIVE  is  the  desire  or  personal  motivation  to  achieve  a  purpose.  Includes  the 
desire  for  self-improvement  or  the  desire  to  manage,  achieve  a  purpose, 
and  improve  output  through  the  teamed  effort  of  subordinates.  Includes 
conscientious,  energetic,  enthusiasm,  forceful,  and  interest. 


EXAMPLES: 

He  is  an  exceptionally  dedicated 
and  hard  working  Chief,  readily 
accepting  and  expeditiously 
solving  problems. 

DRIVE  5 

DRIVE  5 

RELIABILITY  AND  DEPENDABILITY  A 
CONTROLLING  A 

Always  works  to  his  fullest 
potential. 

DRIVE  5 

This  highly  energetic  and  enthusi¬ 
astic  young  petty  officer  has  per¬ 
formed  all  aspects  of  his  duties 
in  an  outstanding  fashion. 

DRIVE  A 

DRIVE  A 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

Ratee  performs  with  vigor. 

DRIVE  3 

He  is  conscientious  and  is  always 
extremely  dependable. 

DRIVE  3 

RELIABILITY  AND  DEPENDABILITY  5 

Although  the  tasks  assigned  to 
the  personnel  under  his  supervi¬ 
sion  are  always  completed  effi¬ 
ciently  and  in  an  excellent  to 
outstanding  manner,  his  superiors 
feel  that  he  is  capable  of  getting 
more  out  of  his  men  and  of  putting 
more  of  himself  into  the  job. 

PRODUCTIVITY  AND  ACHIEVEMENT  5 
LEADERSHIP  AND  DIRECTING  2 

DRIVE  2 

Ratee  is  not  as  aggressive  as  he 
could  be  which  subsequently  de¬ 
tracts  from  him  reaching  his  full 
potential. 

DRIVE  2 

POTENTIAL  2 

Additional  Indexing  Clues: 

/ 

He  continually  strives  for  per¬ 
fection. 

DRIVE  5 

Always  driving  to  do  the  best 
job  possible 

DRIVE  5 

Completely  dedicated 

DRIVE  A 

Attempts  to  achieve  perfection 

DRIVE  A 

Volunteers  for  jobs 

DRIVE  3 
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DRIVE  (Continued) 

The  following  words  and  phrases  would  be  indexed  as  DRIVE  4:  unflagging 
effort;  works  very  hard  at  his  job;  spends  many  extra  hours  at  his  job; 
dedicates  or  devotes  long  hours;  tireless;  chief  is  tireless  in  his  ef¬ 
forts  to  improve  his  equipment,  personnel,  and  records;  always  willing 
to  work  long  hours  whenever  necessary;  continuously  strives;  taking  courses 
in  off-duty  hours. 

The  following  words  and  phrases  would  be  indexed  as  DRIVE  3:  intent  on 
doing  job  well;  determined;  works  hard;  applied  himself;  strives  for  im¬ 
provement;  drive;  working  long  hours;  aggressive  pursuit  of  duties; 
can-do  spirit  or  attitude;  diligent;  dynamic;  ambitious;  zeal;  endeavor. 

When  a  statement  about  drive  is  part  of  how  a  ratee  is  performing  a 
specific  function  or  skill,  then  the  statement  is  used  as  a  qualifier 
for  that  function  or  skill.  For  example,  "Ratee  works  very  hard  at  moti¬ 
vating  his  men  to  improve  their  performance, "  would  be  indexed  as  LEADER¬ 
SHIP  AND  DIRECTING  4,  and  DRIVE  would  not  be  used.  However,  a  statement 
such  as,  "Ratee  is  constantly  seeking  to  improve  the  productivity  of  the 
work  center,"  would  be  indexed  as  DRIVE  4  since  there  is  no  indication 
that  this  objective  has  been  achieved  and  there  is  no  mention  of  any 
specific  function  or  skill. 
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ENDURANCE  is  the  ability  to  complete  tasks  under  conditions  of  fatigue,  dis 
tress,  stress,  and/or  pain. 

EXAMPLES: 


His  endurance  is  outstanding. 

ENDURANCE  5 

During  the  previous  deployment  he 
demonstrated  his  ability  to  func¬ 
tion  smoothly  and  effectively  under 
adverse  and  demanding  conditions 
for  extended  periods. 

ENDURANCE  4 

His  ability  to  maintain  an  inner 
calm  and  to  function  efficiently 
during  periods  of  great  confusion 
and  stress  suit  him  ideally  to  his 
present  assignment. 

CONDUCT,  INTEGRITY,  AND  PRIDE  3 
ENDURANCE  4 

He  uncomplainingly  responds  when 
called  upon  to  meet  unscheduled 
commitments,  frequently  under 
adverse  conditions. 

RESPONSIVENESS  4 

ENDURANCE  3 

Additional  Indexing  Clues: 


Performs  well  under  stress 

ENDURANCE  4 

His  persistence  (tenacity)  in 
the  face  of  adversity 

ENDURANCE  3 

Fortitude 

ENDURANCE  3 

Indefatigable 

ENDURANCE  3 

Tested  and  proven  under  hostile 
fire 

ENDURANCE  3 

Perseverance 

ENDURANCE  3 
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FLEXIBILITY  is  the  quality  of  being  adjustable  or  adaptable  to  change;  capable 
of  modification. 

EXAMPLES : 


His  keen  mind  is  alert  to  all 
possible  circumstances,  and  he 
succeeds  brilliantly  in  adjusting 
to  new  environments. 

INTELLECTUAL  FUNCTIONING  4 
FLEXIBILITY  5 

Overall,  he  is  a  highly  adaptable 
individual  who  exhibits  unlimited 
potential  and  continuing  high 
value  to  the  U.S.  Navy. 

FLEXIBILITY  4 

POTENTIAL  5 

ASSET  TO  THE  NAVY  4 

He  is  a  mature,  stable  Chief 

Petty  Officer,  intelligent, 
adaptable,  and  reliable. 

RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 
INTELLECTUAL  FUNCTIONING  3 
FLEXIBILITY  3 

RELIABILITY  AND  DEPENDABILITY  3 

Additional  Indexing  Clues: 


Adjusts  quickly 

FLEXIBILITY  4 

Able  to  handle  varied  jobs  and 
hence  is  a  very  valuable  Chief 

FLEXIBILITY  3 

ASSET  TO  THE  NAVY  4 

Flexible 

FLEXIBILITY  3 

Open  minded 

FLEXIBILITY  3 

Open  mind  to  criticism 

FLEXIBILITY  3 
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GROOMING  AND  ATTIRE  is  the  way  in  which 
appearance  and  clothing, 

EXAMPLES : 

Neat  and  polished  appearance  is  in 
keeping  with  the  highest  Navy  stan¬ 
dards  and  serves  as  a  criteria  of 
excellence  among  the  men  with  whom 
he  comes  in  contact. 

Ratee’s  personal  appearance  is 
always  correct  and  proper. 

He  is  always  neat  in  appearance 
and  his  conduct  is  exemplary. 

He  is  tidy,  intelligent,  and  ob¬ 
tains  the  best  results  from  his 
men. 

His  tendency  towards  being  over¬ 
weight  greatly  detracts  from  his 
overall  appearance. 

Additional  Indexing  Clues: 

His  impeccable  appearance  leaves 
nothing  to  be  desired. 

Wearing  of  uniform  excellent 

Always  impeccable 

His  appearance  is  immaculate 
at  all  times. 

His  appearance  and  dress  is  always 
correct,  smart,  and  impressive. 

His  uniform  is  consistently 
immaculate. 

His  appearance  is  good  and  sets 
an  example  for  all. 

His  dress  is  impressive,  and  is 
worn  with  care. 

Uniform  (or  dress)  immaculate 

Gr o  oming  imp  ec  c  ab 1 e 

He  takes  pride  in  his  appear¬ 
ance. 

Attention  to  his  appearance 
Appearance  military 
Shined  shoes 


person  cares  for  his  physical 


GROOMING  AND  ATTIRE  5 


GROOMING  AND  ATTIRE  4 

GROOMING  AND  ATTIRE  4 
CONDUCT,  INTEGRITY,  AND  PRIDE  4 

GROOMING  AND  ATTIRE  3 
INTELLECTUAL  FUNCTIONING  3 
LEADERSHIP  AND  DIRECTING  5 

GROOMING  AND  ATTIRE  2 


GROOMING  AND  ATTIRE  5 

GROOMING  AND  ATTIRE  5 
GROOMING  AND  ATTIRE  4 
GROOMING  AND  ATTIRE  4 

GROOMING  AND  ATTIRE  4 

GROOMING  AND  ATTIRE  4 

GROOMING  AND  ATTIRE  4 

GROOMING  AND  ATTIRE  4 

GROOMING  AND  ATTIRE  4 
GROOMING  AND  ATTIRE  4 
GROOMING  AND  ATTIRE  3 

GROOMING  AND  ATTIRE  3 
GROOMING  AND  ATTIRE  3 
GROOMING  AND  ATTIRE  3 
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INITIATIVE  is  the  ability  to  recognize  and  originate  necessary  or  appropriate 
tasks  on  one’s  own  without  being  asked. 

EXAMPLES : 


His  personal  appearance,  cooperative 
attitude,  and  initiative  leave  vir¬ 
tually  nothing  to  be  desired. 

He  is  extremely  reliable  and  never 
fails  to  take  the  initiative  in 
difficult  situations. 

He  displays  an  outstanding  example 
in  initiative. 

Ratee  displays  considerable  initia¬ 
tive  in  accomplishing  each  task 
assigned. 

Ratee  possesses  a  keen  mind,  is 
capable  of  original  thinking,  and 
expresses  his  thoughts  well  and 
decisively  when  communicating 
with  others. 

Ratee  is  proficient  in  anticipat¬ 
ing  situations  in  his  area  of 
responsibility  and  initiates 
action  to  cope  with  it. 

Ratee  undertakes  his  duties  will¬ 
ingly  but  lacks  the  initiative 
that  is  required  of  the  very  best 
in  his  rate. 

However,  at  times  he  lacks  the 
initiative  and  drive  that  is  neces¬ 
sary  to  produce  these  results. 

Additional  Indexing  Clues: 

Provided  Division  officer  with 
many  ideas 

Volunteering  his  own  views 

Making  suggestions  (recommendations) 

An  original  thinker 

Institutes 


GROOMING  AND  ATTIRE  5 
COOPERATION  5 
INITIATIVE  5 

RELIABILITY  AND  DEPENDABILITY  5 
INITIATIVE  5 


INITIATIVE  5 


INITIATIVE  4 


INTELLECTUAL  FUNCTIONING  3 
INITIATIVE  3 
COMMUNICATION  4 


CONTROLLING  4 
INITIATIVE  3 


RELIABILITY  AND  DEPENDABILITY  4 
INITIATIVE  2 


INITIATIVE  2 
DRIVE  2 


INITIATIVE  4 


INITIATIVE  3 
INITIATIVE  3 
INITIATIVE  3 
INITIATIVE  3 


"He  has  shown  initiative  by  repairing  the  probe  refueling  hose  and  fixing 
the  main  feed  pumps  in  the  forward  fireroom.,f  INITIATIVE  3;  PRODUCTIVITY 
AND  ACHIEVEMENT  3.  Here  the  index  term  PRODUCTIVITY  AND  ACHIEVEMENT  is 
used  rather  than  TECHNICAL  SKILLS  because  the  indexer  cannot  be  certain 
whether  the  ratee  had  the  skill  to  do  this  himself  or  whether  he  actually 
managed  or  directed  others  to  accomplish  this  task.  CONTROLLING  is  not 
used  here  because  the  ratee’ s  main  job  duty  was  maintenance. 
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INTELLECTUAL  FUNCTIONING  is  the  ability  to  learn  or  understand  from  experience, 
and  the  ability  to  analyze,  reason,  and  perceive  relationships  and  differ¬ 
ences. 


A  measurement  of  intellectual  functioning  would  be  a  scholastic  record. 
EXAMPLES: 


He  completed  all  courses  with  very 
high  grades,  usually  leading  his 
class. 

Ratee  is  extremely  intelligent. 

He  learns  quickly  and  applies  his 
training  and  experience  effectively. 

He  is  a  forceful  and  intelligent 
Career  Petty  Officer  who  has  such 
command  of  the  basic  professional 
techniques  that  he  can  direct  his 
actions  to  job  perfection  and  the 
well-being  of  his  subordinates. 

He  is  tidy,  intelligent,  and 
obtains  the  best  results  from 
his  men. 

Additional  Indexing  Clues: 

Learned  quickly 

Coherent  mental  organization 

Coherence  of  his  thoughts 

Intelligence 

Keen  minded 

Common  sense 

Judgment 

Insight 


INTELLECTUAL  FUNCTIONING  5 


INTELLECTUAL  FUNCTIONING  5 

INTELLECTUAL  FUNCTIONING  A 
PRODUCTIVITY  AND  ACHIEVEMENT  A 

DRIVE  3 

INTELLECTUAL  FUNCTIONING  3 
PROFESSIONALISM  A 
TECHNICAL  SKILLS  A 
PRODUCTIVITY  AND  ACHIEVEMENT  5 
LEADERSHIP  AND  DIRECTING  A 

GROOMING  AND  ATTIRE  3 
INTELLECTUAL  FUNCTIONING  3 
LEADERSHIP  AND  DIRECTING  5 


INTELLECTUAL  FUNCTIONING  A 
INTELLECTUAL  FUNCTIONING  3 
INTELLECTUAL  FUNCTIONING  3 
INTELLECTUAL  FUNCTIONING  3 
INTELLECTUAL  FUNCTIONING  3 
INTELLECTUAL  FUNCTIONING  3 
INTELLECTUAL  FUNCTIONING  3 
INTELLECTUAL  FUNCTIONING  3 


Alert  would  be  indexed  as  INTELLECTUAL  FUNCTIONING  3.  However,  if  the 
ratee  is  alert  to  a  command  or  an  order,  it  would  be  indexed  as  RESPON¬ 
SIVENESS  3. 


Analytical  mind  would  be  indexed  as  INTELLECTUAL  FUNCTIONING  3.  Some¬ 
times  intellectual  functioning,  planning,  and  controlling  become  confused. 

It  is  important  here  to  keep  in  mind  whether  the  rater  was  referring  to 
the  function  of  planning  or  controlling  or  whether  he  was  making  a  state¬ 
ment  about  the  ratee's  intellectual  functioning,  i.e.,  his  general  analyti¬ 
cal  ability,  his  ability  to  proceed  logically,  or  his  problem-solving  abili¬ 
ty.  For  example,  "His  imagination  allows  him  to  find  new  and  different 
solutions  to  problems  which  others  do  not  seem  to  be  able  to  solve,"  would 
be  indexed  as  RESOURCEFULNESS  3  and  INTELLECTUAL  FUNCTIONING  A. 
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LEADERSHIP  AND  DIRECTING  represent  motivating,  guiding,  and  supervising  of 

subordinates  to  accomplish  a  job  and  work  towards  improved  performance. 
Includes  encouraging  subordinates  in  cooperative  endeavors  and  also  in 
self-development  through  counseling.  A  measure  would  be  morale. 

LEADERSHIP  AND  DIRECTING  also  would  be  creating  an  atmosphere  that  makes 
teamwork  possible,  such  as  improving  working  conditions. 

EXAMPLES: 


His  excellent  leadership  qualities 
were  especially  manifest  when  he 
was  assigned  to  lead  the  mainte¬ 
nance  efforts  on  CQ,  detachments  in 
CONSTELLATION  and  INDEPENDENCE. 

LEADERSHIP  AND  DIRECTING  5 

Ratee  handles  his  men  in  an  effec¬ 
tive  manner  and  always  gets  good 
results  from  his  subordinates. 

LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  4 

Ratee* s  knowledge  of  the  S-2E 
Electronic  System  is  outstanding, 
and  he  is  able  to  utilize  this 
exceptional  knowledge  through 
skillful  management  of  shop  person¬ 
nel  and  an  innate  ability  to  pass 
along  what  he  knows  to  others. 

TECHNICAL  SKILLS  5 

TECHNICAL  SKILLS  5 

LEADERSHIP  AND  DIRECTING  4 
COMMUNICATION  4 

He  spent  a  great  deal  of  extra  time 
with  his  men  and  turned  out  well 
trained,  well  motivated  men  for  the 
fleet. 

LEADERSHIP  AND  DIRECTING  4 
STAFFING  4 

LEADERSHIP  AND  DIRECTING  4 

Ratee  has  the  ability  to  solve  prob¬ 
lems  and  motivate  people. 

INTELLECTUAL  FUNCTIONING  3 
LEADERSHIP  AND  DIRECTING  3 

His  loyalty  was  shown  in  the  con¬ 
scientious  manner  in  which  he  attend¬ 
ed  to  the  problems  of  his  men. 

CONDUCT,  INTEGRITY,  AND  PRIDE  3 
LEADERSHIP  AND  DIRECTING  3 

He  appears  to  be  indifferent  to 
the  personnel  administration  of 
his  men  especially  in  regard  to 
special  requests  and  advancement. 

LEADERSHIP  AND  DIRECTING  2 

Although  the  tasks  assigned  to  the 
personnel  under  his  supervision  are 
always  completed  efficiently  and  in 
an  excellent  to  outstanding  manner, 
his  superiors  feel  that  he  is  capa¬ 
ble  of  getting  more  out  of  his  men 
and  of  putting  more  of  himself  into 
the  job. 

PRODUCTIVITY  AND  ACHIEVEMENT  5 
LEADERSHIP  AND  DIRECTING  2 

DRIVE  2 
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LEADERSHIP  AND  DIRECTING  (Continued) 

Additional  Indexing  Clues: 

Gets  most  out  of  them 

Best  results  from  men 

Consistent  in  direction  of 
personnel 

High  shop  esprit 

High  regard  for  men 

Aggressive  leadership 

Molded  crew  into  competent  and 
effective  team 

Directs  men  well 

High  expectations  from  men 

High  regard  for  team  concept 

Promotes  harmony  and  accord 

Men  cheerful 

Led  the  shop 

Encourages  and  guides  subordinates 

Helped  men  advance  in  rate 

Leads  by  setting  the  example 

Leadership  ability 

Men  trust  him/loyal  to  him 

Finds  time  to  direct  and 
counsel  young  men 

Skilled  at  managing  his  men 

Tact  in  handling  subordinates 

His  men  are  hard  working 

No  disciplinary  problems 

Mindful  of  his  position  as  a 
leader 

Lack  of  leadership 

Needs  more  forceful  approach 
to  leadership 


LEADERSHIP  AND  DIRECTING  5 
LEADERSHIP  AND  DIRECTING  5 
LEADERSHIP  AND  DIRECTING  4 

LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  4 

LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  4 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 

LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 
LEADERSHIP  AND  DIRECTING  3 

LEADERSHIP  AND  DIRECTING  2 
LEADERSHIP  AND  DIRECTING  2 


Many  experts  in  the  field  of  management  feel  that  the  connection  between 
performance  and  possession  of  traits  is  doubtful.  But  practically  every 
study  has  found  successful  managers  to  be  strong  leaders.  Leadership  is 
not  only  an  ability  or  trait  but  it  is  also  a  very  important  function  of 
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LEADERSHIP  AND  DIRECTING  (Continued) 

management.  To  direct  his  subordinates,  a  manager  must  motivate,  com¬ 
municate,  and  lead.  Directing  is  a  function  that  includes  all  those 
activities  which  are  designed  to  encourage  subordinates  to  work  effec¬ 
tively  and  efficiently.  A  manager  must  be  concerned  for  human  feelings 
and  morale.  For  example,  "He  is  tactful  with  his  subordinates."  It  can 
be  seen  here  that  interpersonal  relations  between  a  ratee  and  his  sub¬ 
ordinates  is  important  to  the  leadership  and  directing  function.  The 
rapport  that  a  ratee  has  with  his  divisional  personnel  would  be  indexed 
with  LEADERSHIP  AND  DIRECTING.  However,  a  ratee1 s  interpersonal  rela¬ 
tionship  with  his  peers  or  superiors  probably  would  be  indexed  by  one  of 
the  following  terms:  CONDUCT,  INTEGRITY,  AND  PRIDE;  COOPERATION;  REPRE¬ 
SENTATION;  or  RESPONSIVENESS.  There  is  one  exception  to  this  rule.  When 
the  statement  has  to  do  with  a  ratee1 s  cooperative  attitude  in  working 
with  others,  superiors  or  subordinates,  use  the  index  term  COOPERATION. 

The  rapport  that  a  ratee  has  with  other  organizational  units  enhances  his 
division’s  work  and,  therefore,  the  index  term  REPRESENTATION  would  be 
used . 

A  manager  must  act  his  part  and  be  conscious  of  the  impact  of  his  be¬ 
havior  on  his  men.  For  example,  "He  leads  by  example." 

A  manager  also  must  look  after  the  individual  needs  of  his  subordinates 
and  provide  a  challenge  for  them.  He  must  be  responsive  to  their  needs 
and  to  their  ideas  and  suggestions. 

The  words  supervising  or  directing  can  apply  to  tasks,  men,  or  both.  The 
indexer  has  to  judge  what  the  rater  means.  It  usually  can  be  assumed  that 
supervising  refers  to  the  overall  managerial  functions  or  tasks  (MANAGE¬ 
MENT  FUNCTIONS),  unless  stated  or  inferred  otherwise. 
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MANAGEMENT  FUNCTIONS  are  those  job  duties  which  are  characteristic  of  all 

managers.  Though  operations  may  differ  from  one  organization  to  another, 
the  functions  of  the  manager  are  common  to  all. 

EXAMPLES : 


He  is  an  excellent  manager  and 
organizer  who  is  willing  to  accept 
any  assignment  no  matter  how 
difficult. 

MANAGEMENT  FUNCTIONS  5 

ORGANIZATION  5 

RELIABILITY  AND  DEPENDABILITY  5 

Chief  XX  has  made  a  prime  contribu¬ 
tion  to  the  ship  through  his  excel¬ 
lent  supervision  of  the  Fuel  Oil 
and  Water  Testing  Laboratory  and 
the  Oil  Kings. 

PRODUCTIVITY  AND  ACHIEVEMENT  4 
MANAGEMENT  FUNCTIONS  5 

Ratee’s  superior  leadership  capa¬ 
bilities  and  overall  knowledge  of 
management  greatly  contributed  to 
this  division  receiving  a  grade  of 

4.0  during  the  annual  administra¬ 
tive  inspection. 

LEADERSHIP  AND  DIRECTING  5 
MANAGEMENT  FUNCTIONS  4 

PRODUCTIVITY  AND  ACHIEVEMENT  4 

He  is  well  versed  in  the  3-M  System 
and  always  exhibits  sound  manage¬ 
ment  practices. 

TECHNICAL  SKILLS  4 

MANAGEMENT  FUNCTIONS  4 

His  administrative  knowledge  and 
ability  to  supervise  and  coordinate 
the  efforts  of  other  instructors 
enabled  this  command  to  develop 
all  the  material  required  for 
realistic  support  of  the  E2B 

Aircraft. 

MANAGEMENT  FUNCTIONS  3 

LEADERSHIP  AND  DIRECTING  3 
ORGANIZATION  3 

PRODUCTIVITY  AND  ACHIEVEMENT  4 

BTC  has  an  excellent  working  and 
practical  knowledge  of  the  PMS 

System  but  has  a  tendency  to  be 
lax  in  the  administrative  phase 
of  the  system. 

TECHNICAL  SKILLS  5 

MANAGEMENT  FUNCTIONS  2 

Additional  Indexing  Clues: 


Proficient  Petty  Officer 

MANAGEMENT  FUNCTIONS  4 

The  words  supervising  or  directing  can  apply  to  tasks  or  men.  The  indexer 
has  to  judge  what  the  rater  means.  It  usually  can  be  assumed  that  super¬ 
vising  refers  to  the  overall  managerial  functions  or  tasks,  unless  stated 
or  inferred  otherwise.  If  the  statement  refers  to  the  supervising  or 
directing  of  men  only,  then  use  LEADERSHIP  AND  DIRECTING. 
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MANAGEMENT  FUNCTIONS  CContinued) 

E-7/8/9  Evaluation  Reports  describe  the  on-job  performance  of  junior 
level  managers.  Managers  at  a  junior  level  have  technical  as  well  as 
managerial  functions.  This  should  be  remembered  when  indexing  a  state¬ 
ment  that  includes  the  ratee’ s  job  title.  For  example,  if  the  statement 
claims  that  the  ratee  is  an  ’’excellent  Radioman  or  Boilerman,"  we  would 
assume  that  the  rater  was  referring  to  his  technical  skills  and  label  it 
as  such.  But  on  the  other  hand,  if  a  statement  claims  that  the  ratee 
is  an  Excellent  Chief  Petty  Officer  or  Supervisor, "  it  would  be  assumed 
that  the  rater  is  referring  more  to  the  ratee ’s  managerial  functions  and 
would  be  labeled  accordingly.  Also  remember  that  if  Instructor  is  a 
ratee’ s  job  title,  a  statement  regarding  his  instructing  would  be  labeled 
as  MANAGEMENT  FUNCTIONS,  whereas  if  the  reference  is  to  how  a  ratee  in¬ 
structs  during  the  course  of  his  other  duties,  use  the  label  COMMUNICA¬ 
TION. 

In  evaluating  a  ratee,  the  indexer  should  be  interested  in  how  the  ratee 
performs  his  managerial  functions.  These  functions  are  the  means  by 
which  the  manager  proceeds  to  accomplish  his  job.  When  a  statement  re¬ 
fers  to  how  he  performs  these  overall  tasks,  use  MANAGEMENT  FUNCTIONS. 
However,  when  a  statement  refers  to  the  results  accomplished  from  per¬ 
forming  these  tasks,  use  PRODUCTIVITY  AND  ACHIEVEMENT. 
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ORGANIZATION  is  the  establishment  of  an  Intentional  structure  of  roles  through 
the  determination  and  enumeration  of  activities  required  to  achieve  enter 
prise  goals  such  as  grouping  activities  and  roles,  delegating  authority, 
and  coordinating  authority  relationships. 

EXAMPLES: 


He  does  an  excellent  job  of 
planning,  organizing,  and  carry¬ 
ing  out  his  job. 

PLANNING  5 

ORGANIZATION  5 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

He  has  developed  an  extremely 

ORGANIZATION  5 

tightly  knit  division  which  has  an  LEADERSHIP  AND  DIRECTING  5 

uncommon  amount  of  pride  in  its  work. 


He  is  a  highly  motivated  and 
aggressive  individual  with  a  good 
sense  or  organization  and  adminis¬ 
trative  ability. 

DRIVE  4 

DRIVE  4 

ORGANIZATION  4 

MANAGEMENT  FUNCTIONS  4 

Ratee  spends  a  very  limited  time  in 
the  shop  but  has  exerted  a  spirit  of 
independence  in  his  First  Class,  the 
result  being  a  well  organized  and 
efficient  shop. 

DRIVE  2 

LEADERSHIP  AND  DIRECTING  4 
ORGANIZATION  4 

PRODUCTIVITY  AND  ACHIEVEMENT  4 

His  ability  to  assign  workload  in  a 
smooth  fashion  is  noteworthy. 

ORGANIZATION  4 

He  is  a  professional  administrator 
and  understands  the  principles  of 
delegation. 

PROFESSIONALISM  3 

MANAGEMENT  FUNCTIONS  4 
ORGANIZATION  3 

The  ratee’ s  ability  to  work  with 
others,  his  capacity  for  organi¬ 
zation  and  stimulating  enthusiasm 
makes  him  a  valuable  asset  to  any 
unit. 

COOPERATION  3 

ORGANIZATION  3 

DRIVE  4 

ASSET  TO  THE  NAVY  4 

He  reorganized  the  work  center. 

ORGANIZATION  3 

His  initiative  at  times  lags,  and 
he  tends  to  undertake  too  many 
tasks  alone  rather  than  delegating 
them  to  subordinates. 

INITIATIVE  2 

ORGANIZATION  2 

Additional  Indexing  Clues: 


Excellent  organizer 

ORGANIZATION  5 

Excellent  job  setting  up  the 
operation 

ORGANIZATION  5 
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ORGANIZATION  (Continued) 


Set  up  a  file  on  each  item 


ORGANIZATION  3 


Ability  to  coordinate 


ORGANIZATION  3 


Coordinates  work  centers 


ORGANIZATION  3 


Coordination  of  work  duties 


ORGANIZATION  3 


If  a  ratee  sets  up  a  liaison  with  another  organizational  unit  or  division 
within  the  Navy  or  an  outside  organization,  the  index  term  ORGANIZATION 
would  be  used.  However,  if  a  ratee  uses  the  liaison  or  organizational 
structure  that  is  already  set  up  to  enhance  his  division's  working  rela¬ 
tionship  with  other  organizational  units,  the  index  term  REPRESENTATION 
would  be  used.  If  a  liaison  is  used  as  a  channel  of  communication,  then 
the  index  term  USE  OF  COMMUNICATION  would  be  used. 
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PLANNING  is  a  decision-making  process  involving  the  selection  among  alterna¬ 
tives  of  objectives,  policies,  and  programs  and  the  procedures  for  achiev 
ing  them. 

Types  of  plans  include  objectives,  policies,  methods  or  procedures,  rules 
programs,  budgets,  strategies,  schedules,  traffic  flow  patterns,  and  in¬ 
ventory. 

EXAMPLES : 


He  does  an  excellent  job  of  plan¬ 
ning,  organizing,  and  carrying 
out  his  job. 

He  is  methodical,  deliberate,  and 
able  to  develop  effective  and 
efficient  procedural  methods  of 
accomplishing  the  division  work¬ 
load. 

Ratee's  resourcefulness  in  setting 
up  a  procedure  to  instruct  150 
recruits  daily  in  this  very  diffi¬ 
cult  operation  was  largely  re¬ 
sponsible  for  the  efficient 
operation  of  the  matches. 

He  had  the  ability  and  initiative 
to  plan  and  assign  work  to  person¬ 
nel  under  his  direction  to  ensure 
that  the  end  results  are  of  the 
highest  quality. 

His  analysis  of  divisional  prob¬ 
lems,  both  functional  and  adminis¬ 
trative,  and  the  execution  of 
corrective  measures  have  been 
very  conducive  to  a  smooth  and 
highly  effective  division. 

Additional  Indexing  Clues: 

Found  new  ways  to  improve  the 
security  of  the  Communications 
building 

Good  decision  maker 


PLANNING  5 
ORGANIZATION  5 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

SKILLS  AND  ABILITIES  3 
SKILLS  AND  ABILITIES  3 
PLANNING  4 


RESOURCEFULNESS  3 
PLANNING  4 

PRODUCTIVITY  AND  ACHIEVEMENT  4 


INITIATIVE  3 
PLANNING  3 
ORGANIZATION  3 
CONTROLLING  5 


PLANNING  3 
CONTROLLING  3 

PRODUCTIVITY  AND  ACHIEVEMENT  4 


PLANNING  4 


PLANNING  4 


Improves  plans  PLANNING  4 

Has  alternate  plans  PLANNING  3 
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POTENTIAL  refers  to  the  capacity  and/or  recommendation  for  an  individual  to 
assume  a  higher  rank  or  added  responsibilities, 

EXAMPLES : 


Chief  XX  has  demonstrated  the 
potential  to  be  an  outstanding 
leader. 

POTENTIAL  5 

He  would  be  most  valuable  in  an 
instructor  billet  assignment. 

POTENTIAL  5 

There  is  nothing  that  can  arise  in 
his  present  position  or  next  assign¬ 
ment  that  ratee  cannot  cope  with. 

POTENTIAL  5 

He  is  highly  recommended  for 
advancement. 

POTENTIAL  4 

Chief  XX  is  highly  recommended  for 
advancement  and  retention  in  the 

Naval  Reserve, 

POTENTIAL  4 

He  is  qualified  to  assume  the 
greater  responsibility  of  the 
next  higher  pay  grade. 

SKILLS  AND  ABILITIES  3 
POTENTIAL  3 

Ratee  is  fully  qualified  for 
advancement  in  rate. 

SKILLS  AND  ABILITIES  4 
POTENTIAL  3 

Ratee  is  very  knowledgeable  in  the 
supply  system  and  is  recommended 
for  E-8. 

TECHNICAL  SKILLS  4 

POTENTIAL  3 

With  more  time  and  conscientious 
effort,  he  should  realize  a 
greater  potential. 

POTENTIAL  2 

Additional  Indexing  Clues: 


Outstanding  potential  for  added 
responsibility 

POTENTIAL  5 

Highly  recommended  for  advancement 

POTENTIAL  4 

Strongly  recommended  for  promotion 

POTENTIAL  4 

Eminently  well  qualified  for 
advancement 

SKILLS  AND  ABILITIES  4 
POTENTIAL  3 

Expected  to  improve 

POTENTIAL  3 

Capable  of  assuming  more  responsi¬ 
bility 

POTENTIAL  3 
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PRODUCTIVITY  AND  ACHIEVEMENT  Refers  to  the  successful  accomplishment  of  a 

desired  result  in  terms  of  an  individual's  performance  of  his  job  duties 
(managerial  functions).  Use  this  descriptor  term  unless  a  specific 
function  is  mentioned  in  the  narrative  content  of  the  Evaluation  Report. 

EXAMPLES: 


Every  assignment  is  performed  with 
unusual  accuracy  and  effectiveness. 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

Ratee's  performance  is  outstanding 
in  all  aspects. 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

Performance  in  both  his  primary  and 
secondary  billets  has  been  outstand¬ 
ing  in  all  aspects  as  marked  in 
block  19. 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

Under  his  supervision  the  shop  has 
met  and  surpassed  the  accepted 
norms  of  productivity. 

PRODUCTIVITY  AND  ACHIEVEMENT  4 

Production  over  the  past  six  months 
has  been  about  average  for  a  two- 
man  station. 

PRODUCTIVITY  AND  ACHIEVEMENT  3 

His  technical  competence  and  re¬ 
sourcefulness  contributes  to  ships 
in  tending  to  maintain  operational 
readiness . 

TECHNICAL  SKILLS  3 
RESOURCEFULNESS  3 

PRODUCTIVITY  AND  ACHIEVEMENT  3 

His  performance  has  been  commen¬ 
surate  with  his  rate. 

PRODUCTIVITY  AND  ACHIEVEMENT  3 

While  he  was  acting  as  ship's  Oil 
King,  the  ship  witnessed  three 
oil  spills  while  refueling. 

PRODUCTIVITY  AND  ACHIEVEMENT  2 

His  extremely  poor  performance 
of  his  duties  led  directly  to  the 
ship's  boilers  not  being  in  a 
state  of  operational  readiness. 

PRODUCTIVITY  AND  ACHIEVEMENT  1 

Additional  Indexing  Clues: 


Never  allowed  shop  to  wane 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

Outstanding  performance 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

Handled  duties  in  an  outstanding 
manner 

PRODUCTIVITY  AND  ACHIEVEMENT  5 

Impressive  discharge  of  duty 

PRODUCTIVITY  AND  ACHIEVEMENT  4 
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PRODUCTIVITY  AND  ACHIEVEMENT  (Continued) 


Performed  remarkably  well 

Success  in  a  difficult  task 

Assignments  are  completed  or 
completes  assigned  tasks 

Has  seen  these  tasks  through  to 
their  successful  completion 

Effort  brings  a  success 


PRODUCTIVITY  AND  ACHIEVEMENT  A 
PRODUCTIVITY  AND  ACHIEVEMENT  3 
PRODUCTIVITY  AND  ACHIEVEMENT  3 

PRODUCTIVITY  AND  ACHIEVEMENT  3 

PRODUCTIVITY  AND  ACHIEVEMENT  3 


Statements  of  improvement  should  be  indexed  as  PRODUCTIVITY  AND  ACHIEVE 
MENT.  For  example: 


Made  significant  improvements 
Made  improvements 
Enabled  him  to  improve 
Yielded  benefits 
Room  for  improvement 


PRODUCTIVITY  AND  ACHIEVEMENT  A 
PRODUCTIVITY  AND  ACHIEVEMENT  3 
PRODUCTIVITY  AND  ACHIEVEMENT  3 
PRODUCTIVITY  AND  ACHIEVEMENT  3 
PRODUCTIVITY  AND  ACHIEVEMENT  2 


It  is  important  to  make  certain  that  the  achievement  was  not  a  personal 
technical  accomplishment  in  which  case  TECHNICAL  SKILLS  would  be  used. 


If  a  statement  refers  to  the  successful  accomplishment  of  a  maintenance 
effort  and  it  is  not  the  controlling  function  of  ensuring  the  accomplish¬ 
ment  of  a  plan,  but  rather  the  performance  of  a  ratee's  total  job  func¬ 
tion,  use  PRODUCTIVITY  AND  ACHIEVEMENT*  This  can  be  determined  by  look¬ 
ing  at  the  ratee’s  job  title  or  main  job  duty  as  stated  in  Section  AA. 

The  following  modifiers  can  serve  as  clues  to  labeling  a  statement  or  a 
phrase  as  PRODUCTIVITY  AND  ACHIEVEMENT:  contributed  greatly  to  (A), 
contributed  immensely  to  (A),  significant  increase  in  (A),  contributed 
materially  to  (A),  directly  responsible  for  (3),  directly  contributed 
to  (3),  has  been  instrumental  in  (3),  directly  instrumental  in  (3). 

If  a  statement  of  personal  achievement  (e.g.,  advancement  in  rate)  is 
made  in  a  matter-of-fact  manner,  do  not  index  it*  However,  if  it  is 
stated  as  an  accomplishment,  use  PRODUCTIVITY  AND  ACHIEVEMENT.  For 
example,  "Ratee’s  performance  in  the  past  was  responsible  for  his  selec¬ 
tion  from  every  first  class  and  chief  petty  officer  in  this  command  as 
Command  Career  Counselor.1'  PRODUCTIVITY  AND  ACHIEVEMENT  5. 


An  ability  in  performing  a  task  would  be  indexed  as  PRODUCTIVITY  AND 
ACHIEVEMENT  because  it  tells  how  the  ratee  is  performing  and,  therefore, 
is  a  type  of  measure  of  his  performance.  However,  an  ability  or  skill  to 
perform  in  a  certain  way  suggests  a  future  performance  and  would  be  index¬ 
ed  as  SKILLS  AND  ABILITIES.  For  example,  "His  thoroughness  in  performing 
his  duty  is  outstanding,"  PRODUCTIVITY  AND  ACHIEVEMENT  5;  "Outstanding  in 
his  work,"  PRODUCTIVITY  AND  ACHIEVEMENT  5;  "Chief  XX  is  a  performer,"  PRO¬ 
DUCTIVITY  AND  ACHIEVEMENT  3;  "He  performs  in  a  capable  manner,"  PRODUCTIVI¬ 
TY  AND  ACHIEVEMENT  3;  "He  has  the  ability  to  perform  in  an  outstanding  man¬ 
ner,"  SKILLS  AND  ABILITIES  5. 
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PROFESSIONALISM  is  the  quality  of  being  worthy  of  the  high  standards  of  the 
managerial  profession  and  haying  much  experience  and  great  skill  in  this 
role.  Use  only  when  the  word  professional  or  professionalism  is  applied 
to  the  ratee  and/or  his  performance  and  only  when  it  is  mentioned  specifi¬ 
cally  in  the  narrative  content  of  the  Evaluation  Report. 

EXAMPLES : 


Ratee  has  consistently  demonstrated 
his  outstanding  professional  quali¬ 
fications. 

These  accomplishments  were  achieved 
through  high  professionalism,  en¬ 
thusiasm,  and  superior  decision¬ 
making  ability. 

As  an  Airborne  Communications 
Supervisor,  ratee fs  professional 
competence  has  contributed  to 
flight  operations,  excellent 
mission  performance,  and  praise 
for  his  division. 

His  performance  reflects  compe¬ 
tence  and  professionalism. 

Chief  XX  was  relieved  of  his  duties 
as  the  ship’s  Oil  King  after  serving 
in  the  capacity  for  approximately 
two  months.  He  was  removed  from 
this  billet  because  of  his  lack  of 
professional  knowledge  and  technical 
know-how  in  the  art  of  refueling. 

Additional  Indexing  Clues: 

Professional  attitude 

Professional  behavior 
Professional  administrator 
Professional  skill 

When  the  word  professional  modifies  a 
it  automatically  gives  a  weight  of  4 


PROFESSIONALISM  5 


PROFESSIONALISM  4 
DRIVE  4 
PLANNING  5 


PROFESSIONALISM  3 
PRODUCTIVITY  AND  ACHIEVEMENT  5 


PRODUCTIVITY  AND  ACHIEVEMENT  3 
SKILLS  AND  ABILITIES  3 
PROFESSIONALISM  3 

AWARDS  AND  PUNISHMENT  1 
PROFESSIONALISM  2 
TECHNICAL  SKILLS  2 


PROFESSIONALISM  3 

CONDUCT,  INTEGRITY,  AND  PRIDE  4 

PROFESSIONALISM  3 

CONDUCT,  INTEGRITY,  AND  PRIDE  4 

PROFESSIONALISM  3 
MANAGEMENT  FUNCTIONS  4 

PROFESSIONALISM  3 

specific  skill,  ability  or  function, 
o  that  skill,  ability  or  function. 


When  the  word  professional  modifies  skill  or  ability,  use  only  the  index 
term  PROFESSIONALISM  since  this  term  is  more  specific  than  SKILLS  AND 
ABILITIES  and  falls  under  SKILLS  AND  ABILITIES  in  the  hierarchy  of  index 
terms. 


RELIABILITY  AND  DEPENDABILITY  is  the  combined  quality  of  being  counted  upon  to 
do  what  is  expected  or  required  without  direct  supervision  or  monitoring. 
Connotes  levelheadedness  or  steadiness;  includes  trustworthy ,  responsible , 
constancy ,  stability ,  and  mature . 

EXAMPLES: 


He  is  completely  reliable  and  has 
never  showed  case  by  action  or 
word  to  the  contrary. 

He  is  conscientious  and  is  always 
extremely  dependable. 

Fully  realizes  his  responsibili¬ 
ties  and  at  all  times  consciously 
acts  to  fulfill  them. 

Ratee  is  very  reliable. 

Ratee  is  punctual  and  can  be  de¬ 
pended  upon  to  perform  well  re¬ 
gardless  of  the  amount  of  super¬ 
vision. 

He  can  be  depended  upon  to  "get 
the  job  done"  with  a  minimum  of 
supervision. 

Ratee fs  inability  to  satisfactorily 
discharge  his  financial  obligations 
shows  a  lack  of  responsibility. 

Additional  Indexing  Clues: 

Always  ready  to  accept  additional 
responsibility 

Willing  to  assume  or  accept  added 
responsibility 

Works  well  on  his  own 

Completely  reliable 

Job  done  without  supervision 

Can  work  without  direction 


RELIABILITY  AND  DEPENDABILITY  5 

DRIVE  3 

RELIABILITY  AND  DEPENDABILITY  5 

RELIABILITY  AND  DEPENDABILITY  4 
RELIABILITY  AND  DEPENDABILITY  4 

RELIABILITY  AND  DEPENDABILITY  4 

CONDUCT,  INTEGRITY,  AND  PRIDE  3 
RELIABILITY  AND  DEPENDABILITY  4 

RELIABILITY  AND  DEPENDABILITY  3 

CONDUCT,  INTEGRITY,  AND  PRIDE  2 
RELIABILITY  AND  DEPENDABILITY  2 

RELIABILITY  AND  DEPENDABILITY  5 

RELIABILITY  AND  DEPENDABILITY  4 

RELIABILITY  AND  DEPENDABILITY  4 
RELIABILITY  AND  DEPENDABILITY  4 
RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 


Requires  no  supervision  RELIABILITY  AND  DEPENDABILITY  3 

Dependable  RELIABILITY  AND  DEPENDABILITY  3 
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RELIABILITY  AND  DEPENDABILITY  (Continued) 
Stable 
Trustworthy 
Responsible 
Constancy 

Does  not  need  to  be  supervised 

Performs  without  supervision 

Accepted  responsibilities  and 
authority 

Assumes  added  responsibilities 
Mature 

Reluctance  to  assume  new  tasks 

Need  for  direction  and  checkup 
by  superiors 

Relinquishes  responsibility 


RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 

RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  3 
RELIABILITY  AND  DEPENDABILITY  2 
RELIABILITY  AND  DEPENDABILITY  2 

RELIABILITY  AND  DEPENDABILITY  2 
indexing  that  the  concept  of  responsibili 


It  is  important  to  remember  when 
ty  belongs  under  RELIABILITY  AND  DEPENDABILITY. 
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REPRESENTATION  is  the  creating  of  an  image  of  an  organization  to  the  external 
or  internal  environment. 

EXAMPLES: 


He  is  an  outstanding  representative 
of  the  Navy  in  all  respects, 

Ratee  has  been  very  successful  in 
improving  his  Branch  Station’s 
relations  with  local  high  schools 
and  youth  organizations. 

He  takes  an  active  part  in  church 
and  civic  programs,  helping  to 
uphold  the  Navy  image  in  the 
community. 

The  appearance  of  this  building  is 
a  fine  example  and  reflection  of 
the  Navy  to  the  visiting  civilians. 


REPRESENTATION  5 

REPRESENTATION  4 


REPRESENTATION  4 


REPRESENTATION  4 


Additional  Indexing  Clues: 

Active  in  extracurricular  activities  REPRESENTATION  4 

Takes  part  in  extracurricular  REPRESENTATION  3 

activities 

Public  relations  for  his  branch  REPRESENTATION  3 

Status  of  his  branch  or  job  area  REPRESENTATION  3 

Remember  that  if  a  ratee1 s  job  title  or  main  job  duty  is  public  relations, 
then  the  term  PRODUCTIVITY  AND  ACHIEVEMENT  would  be  used  rather  than 
REPRESENTATION  since  the  statement  would  be  referring  to  the  performance 
of  his  total  job  function. 

Some  management  experts  do  not  consider  REPRESENTATION  to  be  a  separate 
managerial  function.  However,  in  this  dictionary  it  is  considered  to  be 
a  separate  function  because  of  the  importance  given  to  relating  to  the 
civilian  community  by  Naval  personnel. 

If  a  ratee  sets  up  a  liaison  with  another  organizational  unit  or  division 
within  the  Navy  or  an  outside  organization,  the  index  term  ORGANIZATION 
would  be  used.  However,  if  a  ratee  uses  the  liaison  or  organizational 
structure  that  is  already  set  up  to  enhance  his  division’s  working  rela¬ 
tionship  with  other  organizational  units,  the  index  term  REPRESENTATION 
would  be  used.  If  a  liaison  is  used  as  a  channel  of  communication,  then 
the  index  term  USE  OF  COMMUNICATION  would  be  used.  The  rapport  that  a 
ratee  has  with  other  organizational  units  also  enhances  his  division’s 
work,  and  the  index  term  REPRESENTATION  would  be  used.  The  rapport  that 
a  ratee  has  with  his  divisional  personnel  would  be  indexed  with  LEADER¬ 
SHIP  AND  DIRECTING. 
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REPUTE  is  the  position  or  standing  of  an  individual  in  his  relationships  to 
subordinates,  peers,  and  superiors. 

EXAMPLES: 


He  possesses  the  character,  person- 
ality,  and  desire  that  commands 
the  highest  respect  and  admiration 
from  his  associates. 

This  action  has  generated  the 
highest  degree  of  trust  and  con¬ 
fidence  with  his  superiors. 

He  readily  obtains  the  confidence 
of  all  who  come  in  contact  with  him. 

He  is  well  liked  and  highly  re¬ 
spected  by  juniors  and  seniors 
alike,  and  his  easy-going  manner 
and  pleasant  congeniality  make 
him  at  home  in  any  surroundings. 

Ratee  is  highly  regarded  by  all 
of  his  men. 


CONDUCT,  INTEGRITY,  AND  PRIDE  3 
CONDUCT,  INTEGRITY,  AND  PRIDE  3 
DRIVE  3 
REPUTE  5 

REPUTE  5 


REPUTE  4 


REPUTE  4 

CONDUCT,  INTEGRITY,  AND  PRIDE  3 
COOPERATION  4 
FLEXIBILITY  3 


REPUTE  4 


He  expresses  himself  clearly  and  COMMUNICATION  3 

logically  and  his  views  are  re-  REPUTE  3 

spected  by  those  with  whom  he  works. 

Additional  Indexing  Clues: 

He  has  their  full  respect  and  gets  REPUTE  4 

it. 


He  is  highly  regarded  by  his  men. 
Gains  the  genuine  respect 
Praised 


REPUTE  4 
REPUTE  4 
REPUTE  4 


Inspires  respect  REPUTE  3 

Justified  others’ confidence  in  him  REPUTE  3 


Commands  respect  and  gets  it 


REPUTE  3 


Expertise  widely  acknowledged  TECHNICAL  SKILLS  4 

and  respected  by  others  REPUTE  3 


His  advice  is  sought  by  others  REPUTE  3 

because  of  his  technical  skill.  TECHNICAL  SKILLS  3 
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REPUTE  (Continued) 


Merits  the  respect 


REPUTE  3 


Generates  respect 


REPUTE  3 


Respected  by 


REPUTE  3 


Popular 


REPUTE  3 


Many  of  the  feelings  that  a  ratee's  peers  and  superiors  have  for  him 
would  be  Indexed  as  REPUTE  (e.g.,  being  liked  or  trusted  by  his  superi¬ 
ors).  However,  if  it  is  the  ratee's  subordinates  who  have  these  same 
feelings,  it  would  indicate  the  ratee's  leadership  abilities  and  be  la¬ 
beled  as  such  (e.g.,  being  liked  or  trusted  by  his  men).  If  the  words 
respect  or  regard  are  used,  then  automatically  label  this  statement  with 
REPUTE.  For  example,  "His  concern  for  his  men  has  won  the  respect  of 
his  subordinates."  LEADERSHIP  AND  DIRECTING  3;  REPUTE  3. 

The  concept  of  a  ratee's  repute  may  appear  in  the  narrative  text  under 
the  guise  of  one  or  more  descriptive  phrases,  but  this  segment  of  text 
should  be  treated  as  containing  a  single  concept  and  be  labeled  only  once 
with  REPUTE.  For  example,  "He  is  well  liked  and  highly  respected,"  would 
be  treated  as  one  phrase  or  concept  and  would  be  indexed  only  once  with 
the  index  term  REPUTE. 


143 


RESOURCEFULNESS  is  the  innovative  or  creative  ability  to  make  effective  and 
efficient  use  of  men  and  materials  resulting  in  improved  performance. 

EXAMPLES: 


He  has  consistently  demonstrated 
exceptional  resourcefulness  in 
dealing  with  especially  complicated 
projects. 

RESOURCEFULNESS  5 

He  makes  the  best  use  of  all  the 
resources  at  his  command. 

RESOURCEFULNESS  5 

A  direct  indication  of  the  Ratee’s 
resourcefulness  is  indicated  in 
the  fact  that  he  insists  on  obtain¬ 
ing  maximum  utilization  of  the 

Navy’s  training  material. 

RESOURCEFULNESS  3 
RESOURCEFULNESS  5 

Since  the  turnover  of  these  modules 
is  high,  this  represents  a  signifi¬ 
cant  savings  to  the  Navy’s  resources. 

RESOURCEFULNESS  4 

He  has  demonstrated  a  remarkable 
ability  for  substituting  compo¬ 
nents  or  test  equipment  that  might 
be  temporarily  unavailable. 

RESOURCEFULNESS  4 

In  his  area  of  responsibility  he 
requires  that  consumable  modules 
be  repaired  and  reused. 

RESOURCEFULNESS  3 

Chief  XX  is  a  resourceful  man  and 
his  potential  is  great  to  become 
an  even  better  Chief. 

RESOURCEFULNESS  3 

POTENTIAL  4 

His  imagination  allows  him  to  find 
new  and  different  solutions  to 
problems  which  others  do  not  seem 
to  be  able  to  solve. 

RESOURCEFULNESS  3 

INTELLECTUAL  FUNCTIONING  4 

Additional  Indexing  Clues: 


Always  resourceful 

RESOURCEFULNESS  4 

Fully  utilizes  his  men 

RESOURCEFULNESS  4 

Inventiveness 

RESOURCEFULNESS  3 

Imagination 

RESOURCEFULNESS  3 

Ingenuity 

RESOURCEFULNESS  3 
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RESPONSIVENESS  is  the  quality  of  reacting  promptly,  readily,  accurately,  and 
alertly  to  suggestion,  instruction,  or  orders. 

EXAMPLES: 


Ratee  is  extremely  cooperative  in 
all  his  undertakings  when  given 
any  task,  and  completes  it  in  the 
most  expeditious  manner. 

COOPERATION  5 

RESPONSIVENESS  5 

He  is  quick  to  respond  to  any 
situation  or  problem  and  to  find 
a  solution. 

RESPONSIVENESS  4 

INTELLECTUAL  FUNCTIONING  3 

He  uncomplainingly  responds  when 
called  upon  to  meet  unscheduled 
commitments,  frequently  under 
adverse  conditions. 

RESPONSIVENESS  4 

ENDURANCE  3 

He  is  responsive  and  agreeable  to 
demands  upon  him  and  constantly 
seeks  ways  of  improving  working 
conditions  and  morale. 

RESPONSIVENESS  3 

COOPERATION  3 

LEADERSHIP  AND  DIRECTING  4 

Additional  Indexing  Clues: 

Alertness  is  usually  considered  to  be  a  part  of  intellectual  functioning 
However,  if  a  statement  refers  to  a  ratee’ s  alertness  to  an  order,  then 
consider  it  to  be  a  responsive  gesture  on  the  part  of  the  ratee. 

When  in  doubt  about  whether  a  statement  constitutes  a  cooperative  or  re¬ 
sponsible  attitude  as  opposed  to  responsiveness  on  the  part  of  the  ratee 
use  either  COOPERATION  or  RELIABILITY  AND  DEPENDABILITY  rather  than  RE¬ 
SPONSIVENESS. 

Always  use  the  index  term  RESPONSIVENESS  whenever  the  word  responds  is 
used  in  a  statement. 

The  time  element  is  an  important  clue  to  the  use  of  this  term  and,  there 
fore,  would  be  used  to  label  the  following  examples: 


In  a  minimum  of  time 

/ 

RESPONSIVENESS  5 

He  carries  out  all  assignments 
expediently  and  efficiently. 

RESPONSIVENESS  4 

PRODUCTIVITY  AND  ACHIEVEMENT  4 

He  meets  his  responsibilities 
and  quotas  in  a  timely  manner. 

RESPONSIVENESS  3 

Ratee  presented  material  to  the 
students  in  an  expeditious  manner. 

RESPONSIVENESS  3 

RESPONSIVENESS  3 
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SERVICE  MOTIVATION  refers  to  a  serviceman's  contentment  with  life  in  the  Navy, 
or  to  his  desire  or  lack  of  desire  to  fulfill  his  commitment  to  his  job 
duties  in  the  Navy. 

EXAMPLES: 


He  is  extremely  dedicated  to  his 
division,  department,  and  ship. 

SERVICE  MOTIVATION  5 

He  is  an  active  Navy  promoter  and 
is  the  first  to  step  forward  to 
educate  subordinates  on  Navy  policy, 
procedures,  or  benefits. 

SERVICE  MOTIVATION  4 

SERVICE  MOTIVATION  4 

Ratee  always  wears  his  uniform  with 
pride,  both  on  and  off  the  ship. 

SERVICE  MOTIVATION  4 

He  exhibits  pride  in  being  part  of 
this  command  and  is  a  definite 
asset  to  the  Naval  service. 

SERVICE  MOTIVATION  3 

ASSET  TO  THE  NAVY  4 

He  is  a  dedicated  career  man  who 
displays  pride  in  the  Navy  and  the 
Squadron, 

SERVICE  MOTIVATION  3 

SERVICE  MOTIVATION  3 

Additional  Indexing  Clues: 


The  ratee  always  has  the  best 
interests  of  the  Navy  in  mind. 

SERVICE  MOTIVATION  5 

Pride  in  the  Navy 

SERVICE  MOTIVATION  3 

Pride  in  his  unit 

SERVICE  MOTIVATION  3 

Dedication  or  devotion  to  duty 

SERVICE  MOTIVATION  3 

Loyal  to  duty  and  men 

SERVICE  MOTIVATION  3 

Enjoying  his  work 

SERVICE  MOTIVATION  3 
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SKILLS  AND  ABILITIES  are  those  qualities  that  influence  the  producing  of  a 
desired  result  in  an  efficient  and  effective  manner. 

EXAMPLES : 


He  is  able  to  direct  the  efforts  of 
Line  Personnel  in  an  efficient  and 
effective  manner;  this  is  reflected 
in  the  Ratee  by  a  multiple  of  ex¬ 
ceptional  qualities. 

This  man  is  extremely  competent. 

Above  average  in  all  areas,  he  is 
fully  qualified  to  assume  the 
greater  responsibility  of  the  next 
higher  rank. 

He  daily  demonstrates  all  of  the 
highly  desirable  traits  of  a  Chief 
Petty  Officer. 

Ratee  is  very  meticulous  and 
thorough. 

His  natural  abilities  and  responsi¬ 
ble  approach  to  recruiting  have 
enabled  the  ratee  to  outperform 
his  contemporaries. 

Additional  Indexing  Clues; 

Exceptionally  well  qualified 
Outstanding  individual 


LEADERSHIP  AND  DIRECTING  4 
SKILLS  AND  ABILITIES  5 


SKILLS  AND  ABILITIES  5 

SKILLS  AND  ABILITIES  4 
SKILLS  AND  ABILITIES  4 
POTENTIAL  3 


SKILLS  AND  ABILITIES  4 


SKILLS  AND  ABILITIES  4 
SKILLS  AND  ABILITIES  4 

SKILLS  AND  ABILITIES  3 
RELIABILITY  AND  DEPENDABILITY  3 
PRODUCTIVITY  AND  ACHIEVEMENT  4 


SKILLS  AND  ABILITIES  5 
SKILLS  AND  ABILITIES  5 


Well-qualified  Supervisor  SKILLS  AND  ABILITIES  4 

Efficient  SKILLS  AND  ABILITIES  4 

Any  mention  of  managerial  or  administrative  abilities  would  be  labeled 
MANAGEMENT  FUNCTIONS. 


SKILLS  AND  ABILITIES  is  used  as  a  catchall  term  for  those  skills  and 
abilities  which  do  not  fit  any  of  the  more  specific  index  terms  that 
come  under  SKILLS  AND  ABILITIES  in  the  hierarchy.  The  following  terms 
would  be  labeled  under  SKILLS  AND  ABILITIES  since  there  is  no  other  more 
specific  index  term  that  would  describe  them:  qualities,  skills,  traits, 
attention  to  detail,  meticulous,  thorough,  methodical,  competent,  capable 
decisiveness,  deliberate  in  thinking  (deliberate  in  action  would  be  index 
ed  as  CONDUCT,  INTEGRITY,  AND  PRIDE),  and  talented. 
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STAFFING  is  the  manning  of  and  keeping  manned  the  positions  provided  for  by 
the  organization  structure.  Includes  training  and  the  appraisal  or 
assessment  of  personnel . 

EXAMPLES: 


His  instructions  are  presented  in  COMMUNICATION  3 

such  a  manner  that  maximum  training  STAFFING  5 
is  accomplished  in  the  time  alloted. 


He  spent  a  great  deal  of  extra  time  LEADERSHIP  AND  DIRECTING  4 

with  his  men  and  turned  out  well  STAFFING  4 

trained,  well  motivated  men  for  the  LEADERSHIP  AND  DIRECTING  4 

fleet. 


He  is  continually  researching  the  STAFFING  4 

available  training  and  ensuring  CONTROLLING  4 

that  assigned  personnel  have  what 
is  needed. 


Ratee  has  tutored  division  person-  STAFFING  3 

nel  in  all  aspects  of  the  Navy 
publications  system. 

Additional  Indexing  Clues: 

Men  re-enlist  STAFFING  3 

The  manning  of  and  keeping  manned  the  job  positions  would  involve  selec¬ 
tion,  training,  and  the  compensation  of  personnel  in  order  to  keep  them 
in  the  Navy,  It  is  important  to  remember  that  if  the  main  job  duty  is 
recruiting,  then  usually  a  statement  about  manning  of  and  keeping  manned 
positions  in  the  Navy  would  be  the  result  of  a  ratee fs  total  management 
function,  and  hence  would  be  indexed  as  PRODUCTIVITY  AND  ACHIEVEMENT. 
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TECHNICAL  SKILLS  refer  to  the  understanding  and  demonstration  of  techniques 
which  one  applies  to  a  task. 

EXAMPLES: 


The  ratee  has  an  outstanding 
knowledge  of  all  Electrical  Drone 
Systems  in  the  squadron  aircraft. 

TECHNICAL  SKILLS  5 

BTC  has  an  excellent  working  and 
practical  knowledge  of  the  PMS 
system  but  has  a  tendency  to  be 
lax  in  the  administrative  phase 
of  the  system. 

TECHNICAL  SKILLS  5 

MANAGEMENT  FUNCTIONS  2 

He  is  well  versed  in  the  3-M  System 
and  always  exhibits  sound  manage¬ 
ment  practices. 

TECHNICAL  SKILLS  4 

MANAGEMENT  FUNCTIONS  4 

Ratee  single-handedly  tore  down 
and  rebuilt  an  engine  in  16  hours 
so  the  ship  would  be  ready  for  the 
cruise. 

TECHNICAL  SKILLS  4 

He  is  a  very  knowledgeable  techni¬ 
cian  who  attempts  to  keep  constantly 
abreast  of  current  changes  by  de¬ 
voting  his  off  duty  time  to  the 
study  of  technical  and  nontechnical 
material  through  evening  college 
courses . 

TECHNICAL  SKILLS  4 

DRIVE  5 

His  technical  competence  and  re¬ 
sourcefulness  contribute  to  ships 
in  tending  to  maintain  operational 
readiness . 

TECHNICAL  SKILLS  3 
RESOURCEFULNESS  3 

PRODUCTIVITY  AND  ACHIEVEMENT  3 

Chief  XX  was  relieved  of  his  duties 
as  the  ship's  Oil  King  after  serving 
in  the  capacity  for  approximately 
two  months.  He  was  removed  from 
this  billet  because  of  his  lack  of 
professional  knowledge  and  technical 
know-how  in  the  art  of  refueling. 

AWARDS  AND  PUNISHMENT  1 
PROFESSIONALISM  2 

TECHNICAL  SKILLS  2 

Additional  Indexing  Clues: 


Expertise  widely  acknowledged 

TECHNICAL  SKILLS  4 

REPUTE  4 

t  Professional  knowledge 

PROFESSIONALISM  3 

TECHNICAL  SKILLS  4 
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TECHNICAL  SKILLS  (Continued) 

The  following  would  be  indexed  as  TECHNICAL  SKILLS  3:  background,  know¬ 
ledge  of  the  responsibility  of  the  rate,  technical  skills,  technical 
knowledge,  understanding  of  job  duties,  experience,  understanding  of 
work,  "know-how. " 

The  technical  skills  that  a  ratee  possesses  may  be  referred  to  by  the 
evaluator  with  one  or  more  descriptive  phrases,  but  this  part  of  the  sen¬ 
tence  should  be  treated  as  a  single  entity  and  labeled  only  once  with 
TECHNICAL  SKILLS.  For  example,  "...his  lack  of  professional  knowledge 
and  technical  know-how  in  the  art  of  refueling...."  PROFESSIONALISM  2; 
TECHNICAL  SKILLS  2. 

If  the  skill  or  knowledge  is  in  managing  or  in  one  of  the  specific  mana¬ 
gerial  functions,  just  use  MANAGEMENT  FUNCTIONS  or  the  specific  function 
as  the  label.  For  example,  "He  is  skilled  at  managing  his  men,"  would  be 
indexed  as  LEADERSHIP  AND  DIRECTING  3. 

In  the  original  version  of  this  dictionary  there  was  a  term  FUND  OF  KNOW¬ 
LEDGE  which  was  replaced  by  TECHNICAL  SKILLS  since  managers  at  a  junior 
level  have  technical  as  well  as  managerial  functions.  This  is  the  reason 
why  such  phrases  as  "understanding  of  job  duties,  experience,  and  under¬ 
standing  of  work"  are  labeled  as  TECHNICAL  SKILLS.  It  is  important  to 
keep  this  term  in  mind  and  to  discriminate  between  managerial  functions 
and  productivity  or  achievement  as  opposed  to  the  demonstration  of  a 
technical  skill. 

When  a  ratee  is  in  the  process  of  acquiring  new  knowledge  or  a  new  skill, 
it  would  be  an  indication  of  his  drive  and  be  labeled  as  such.  The  in¬ 
dexer  should  not  use  TECHNICAL  SKILLS  unless  the  ratee  has  finished  the 
course  and,  therefore,  already  has  the  knowledge  or  skill. 


150 


USE  OF  COMMUNICATION  is  the  use  an  individual  makes  of  the  opportunities  for 
information  exchange  within  an  organization. 

SEE  ALSO:  COMMUNICATION 

EXAMPLES: 


He  is  an  excellent  administrator 
and  a  skillful  supervisor,  and  he 
keeps  himself  and  his  superiors 
fully  informed  of  all  facets  of 
his  branch’s  operation. 

He  always  keeps  his  superiors 
informed  of  any  problems  and  the 
status  of  work  in  progress, 

Ratee  always  consults  with  his 
Division  Officer  concerning  his 
wishes  on  a  matter,  unless  time 
is  essential. 

He  keeps  his  seniors  alerted  to 
pending  problems. 

Additional  Indexing  Clues: 

Keeps  superior  well  informed 


MANAGEMENT  FUNCTIONS  5 
LEADERSHIP  AND  DIRECTING  4 
USE  OF  COMMUNICATION  5 


USE  OF  COMMUNICATION  5 


USE  OF  COMMUNICATION  4 


USE  OF  COMMUNICATION  3 


USE  OF  COMMUNICATION  4 


Uses  communication  to  arouse  inter-  USE  OF  COMMUNICATION  3 

est,  convince,  and  produce  desired 

results 


This  term  is  used  not  to  indicate  how  a  ratee  communicates,  but  whether 
or  not  he  uses  his  ability  to  communicate  for  the  purpose  of  information 
exchange . 

If  a  ratee  sets  up  a  liaison  with  another  organizational  unit  or  division 
within  the  Navy  or  an  outside  organization,  the  index  term  ORGANIZATION 
would  be  used.  However,  if  a  ratee  uses  the  liaison  or  organizational 
structure  that  is  already  set  up  to  enhance  his  division’s  working  rela¬ 
tionship  with  other  organizational  units,  the  index  term  REPRESENTATION 
would  be  used.  If  a  liaison  is  used  as  a  channel  of  communication,  then 
the  index  term  USE  OF  COMMUNICATION  would  be  used. 


APPENDIX  B 


OUTPUT  FROM  PROGRAM  BMDOID  -  SIMPLE  DATA  DESCRIPTION 
FOR  THE  CROSS  VALIDATION  AND  GENERALIZATION  SAMPLES 


The  output  of  this  computer  program  lists  the 
arithmetic  mean,  the  standard  deviation,  the  standard 
error  of  the  mean,  the  maximum  value,  the  minimum 
value,  and  the  range  for  each  of  the  67  variables 
derived  from  the  content  analysis.  The  output  is 
presented  in  six  parts  for  each  of  the  two  samples. 
First  is  shown  the  output  for  the  Upper,  Middle,  and 
Lower  criterion  groups  on  the  evaluation  section  of 
the  Evaluation  Report.  This  is  followed  by  the  output 
for  the  Upper,  Middle,  and  Lower  criterion  groups  on 
the  justification  section  of  the  Evaluation  Report. 
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CROSS  VALIDATION  SAMPLE  (AT’s  and  BT's) 
UPPER  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

i 

0.3649 

0.5382 

0.0626 

2 

0 

2 

2 

0.1351 

0.5054 

0.0588 

3 

0 

3 

3 

0.5946 

0.9920 

0.1153 

4 

0 

4 

4 

0.0811 

0.2748 

0.0319 

1 

0 

1 

5 

0.0541 

0.2277 

0.0265 

1 

0 

1 

6 

0.0811 

0.3208 

0.0373 

2 

0 

2 

7 

0.0676 

0.2527 

0.0294 

1 

0 

1 

8 

0.0135 

0.1162 

0.0135 

1 

0 

1 

9 

0.4459 

0.6650 

0.0773 

3 

0 

3 

10 

0.1757 

0.4490 

0.0522 

2 

0 

2 

11 

0.6486 

1.1157 

0.1297 

5 

0 

5 

12 

0.0946 

0.2947 

0.0343 

1 

0 

1 

13 

0.1216 

0.3291 

0.0383 

1 

0 

1 

14 

0.0541 

0.2277 

0.0265 

1 

0 

1 

15 

0.0811 

0.3208 

0.0373 

2 

0 

2 

16 

0.1486 

0.3946 

0.0459 

2 

0 

2 

17 

0.2297 

0.6092 

0.0708 

3 

0 

3 

18 

0.2838 

0.5618 

0.0653 

3 

0 

3 

19 

0.2973 

0.6133 

0.0713 

3 

0 

3 

20 

0.0676 

0.2527 

0.0294 

1 

0 

1 

21 

0.0811 

0.4898 

0.0569 

4 

0 

4 

22 

0.6216 

0.9320 

0.1083 

3 

0 

3 

23 

1.3108 

1.2488 

0.1452 

4 

0 

4 

24 

0.2297 

0.4839 

0.0563 

2 

0 

2 

25 

1.1622 

1.4807 

0.1721 

6 

0 

6 

26 

0.2297 

0.5376 

0.0625 

3 

0 

3 

27 

0.3649 

0.5631 

0.0655 

2 

0 

2 

28 

0.2297 

0.6092 

0.0708 

3 

0 

3 

29 

0.0676 

0.2527 

0.0294 

1 

0 

1 

30 

8.2027 

5.4446 

0.6329 

23 

0 

23 

31 

10.9594 

1.4471 

0.1682 

15 

10 

5 

32 

10.2838 

1.0793 

0.1255 

17 

10 

7 

33 

11.2838 

2.1677 

0.2520 

19 

10 

9 

34 

10.1757 

0.6050 

0.0703 

13 

10 

3 

35 

10.0676 

0.3021 

0.0351 

12 

10 

2 

36 

10.1757 

0.6896 

0.0802 

14 

10 

4 

37 

10.1757 

0.6896 

0.0802 

14 

10 

4 

38 

10.0135 

0.1162 

0.0135 

11 

10 

1 

39 

10.8243 

1.2091 

0.1406 

14 

10 

4 

40 

10.3919 

1.0179 

0.1183 

15 

10 

5 

41 

11.3378 

2.6394 

0.3068 

23 

10 

13 

42 

10.2027 

0.6617 

0.0769 

13 

10 

3 

43 

10.2027 

0.6189 

0.0719 

13 

10 

3 

(Continued) 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
UPPER  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 
(Continued) 


No. 

Variable 

Mean 

S.D. 

44 

10.0946 

0.4432 

45 

10.2027 

0.8755 

46 

10.2838 

0.8027 

47 

10.4324 

1.2172 

48 

10.5000 

1.0885 

49 

10.6216 

1.4306 

50 

10.1216 

0.5222 

51 

10.1216 

0.6815 

52 

11.3784 

2.1752 

53 

13.0811 

2.9319 

54 

10.6892 

1.4517 

55 

12.1621 

3.0611 

56 

10.4594 

1.1842 

57 

10.6216 

1.0028 

58 

10.5000 

1.3574 

59 

10.1351 

0.5570 

60 

306.1311 

17.0533 

61 

2.7432 

2.4942 

62 

3.6486 

2.7718 

63 

1.9459 

2.0197 

64 

0.0000 

0.0000 

65 

0.0000 

0.0000 

66 

78.3376 

49.0724 

67 

5.6081 

3.3508 

S.E.  of 


Mean 

Max. 

Min. 

Range 

0.0515 

13 

10 

3 

0.1018 

16 

10 

6 

0.0933 

13 

10 

3 

0.1415 

15 

10 

5 

0.1265 

16 

10 

6 

0.1663 

18 

10 

8 

0.0607 

13 

10 

3 

0.0792 

15 

10 

5 

0.2529 

19 

10 

9 

0.3408 

21 

10 

11 

0.1688 

16 

10 

6 

0.3558 

25 

10 

15 

0.1377 

17 

10 

7 

0.1166 

14 

10 

4 

0.1578 

17 

10 

7 

0.0648 

13 

10 

3 

1.9824 

336 

201 

135 

0.2899 

11 

0 

11 

0.3222 

12 

0 

12 

0.2348 

7 

0 

7 

0.0000 

0 

0 

0 

0.0000 

0 

0 

0 

5.7045 

212 

0 

212 

0.3895 

14 

0 

14 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
MIDDLE  CRITERION  GROUP  (N=74) 

1 9 R- EVALUATION  SECTION 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

i 

0.3108 

0.5471 

0.0636 

2 

0 

2 

2 

0.2162 

0.5040 

0.0586 

2 

0 

2 

3 

1.0000 

1.2928 

0.1503 

4 

0 

4 

4 

0.2297 

0.5376 

0.0625 

2 

0 

2 

5 

0.1216 

0.3291 

0.0383 

1 

0 

1 

6 

0.0811 

0.2748 

0.0319 

1 

0 

1 

7 

0.0676 

0.3021 

0.0351 

2 

0 

2 

8 

0.0405 

0.1986 

0.0231 

1 

0 

1 

9 

0.3919 

0.7552 

0.0878 

4 

0 

4 

10 

0.3108 

0.5951 

0.0692 

3 

0 

3 

11 

1.1892 

1.9492 

0.2266 

9 

0 

9 

12 

0.4595 

0.7437 

0.0865 

4 

0 

4 

13 

0.1081 

0.3126 

0.0363 

1 

0 

1 

14 

0.0946 

0.3380 

0.0393 

2 

0 

2 

15 

0.2297 

0.4235 

0.0492 

1 

0 

1 

16 

0.2162 

0.4145 

0.0482 

1 

0 

1 

17 

0.3649 

0.7134 

0.0829 

3 

0 

3 

18 

0.2162 

0.4463 

0.0519 

2 

0 

2 

19 

0.3514 

0.6066 

0.0705 

2 

0 

2 

20 

0.2568 

0.5748 

0.0668 

3 

0 

3 

21 

0.0405 

0.1986 

0.0231 

1 

0 

1 

22 

0.7162 

0.7855 

0.0913 

3 

0 

3 

23 

0.9189 

0.9029 

0.1050 

3 

0 

3 

24 

0.0946 

0.2947 

0.0343 

1 

0 

1 

25 

1.0135 

1.1526 

0.1340 

5 

0 

5 

26 

0.3378 

0.5558 

0.0646 

2 

0 

2 

27 

0.4459 

0.6853 

0.0797 

3 

0 

3 

28 

0.2432 

0.5442 

0.0633 

2 

0 

2 

29 

0.1486 

0.3582 

0.0416 

1 

0 

1 

30 

10.0811 

5.6052 

0.6516 

25 

0 

25 

31 

10.7027 

1.2576 

0.1462 

15 

10 

5 

32 

10.4054 

0.9497 

0.1104 

14 

10 

4 

33 

11.9189 

2.6881 

0.3125 

19 

10 

9 

34 

10.4054 

0.9640 

0.1121 

14 

10 

4 

35 

10.2297 

0.6313 

0.0734 

12 

10 

2 

36 

10.1486 

0.5150 

0.0599 

12 

10 

2 

37 

10.1351 

0.6688 

0.0777 

15 

10 

5 

38 

10.0811 

0.4303 

0.0500 

13 

10 

3 

39 

10.7297 

1.5641 

0.1818 

20 

10 

10 

40 

10.6081 

1.1799 

0.1372 

16 

10 

6 

41 

11.6351 

2.8649 

0.3330 

22 

5 

17 

42 

10.8919 

1.7405 

0.2023 

20 

10 

10 

43 

10.2432 

0.7367 

0.0856 

13 

10 

3 

(Continued) 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
MIDDLE  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 
(Continued) 


No.  of 

Variable  Mean  S.D. 


44 

10.1351 

0.5811 

45 

10.3919 

0.7907 

46 

10.2838 

0.6086 

47 

10.6621 

1.3677 

48 

10.4324 

1.0347 

49 

10.4459 

0.9529 

50 

10.4054 

0.9054 

51 

10.0405 

0.1986 

52 

11.2973 

1.5055 

53 

12.0270 

2.0672 

54 

10.2838 

0.8840 

55 

11.8243 

2.1155 

56 

10.5540 

0.9951 

57 

10.8784 

1.4893 

58 

10.4054 

0.9351 

59 

10.2703 

0.7639 

60 

308.4282 

10.3687 

61 

2.1216 

2.0604 

62 

4.3784 

3.1740 

63 

3.5270 

2.7259 

64 

0.1892 

1.0813 

65 

0.0000 

0.0000 

66 

86.5808 

42.8689 

67 

7.1621 

3.2011 

S.E.  of 


Mean 

Max. 

Min. 

Range 

0.0675 

13 

9 

4 

0.0919 

13 

10 

3 

0.0708 

13 

10 

3 

0.1590 

17 

10 

7 

0.1203 

16 

10 

6 

0.1108 

14 

8 

6 

0.1052 

13 

10 

3 

0.0231 

11 

10 

1 

0.1750 

17 

10 

7 

0.2403 

17 

10 

7 

0.1028 

13 

10 

3 

0.2459 

20 

10 

10 

0.1157 

14 

10 

4 

0.1731 

18 

10 

8 

0.1087 

14 

10 

4 

0.0888 

13 

9 

4 

1.2053 

337 

289 

48 

0.2395 

12 

0 

12 

0.3690 

13 

0 

13 

0.3169 

11 

0 

11 

0.1257 

9 

0 

9 

0.0000 

0 

0 

0 

4.9834 

214 

0 

214 

0.3721 

15 

0 

15 

ial 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 


CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
LOWER  CRITERION  GROUP  (N-74) 

19R- EVALUATION  SECTION 


S.E.  of 


Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

0.2703 

0.5047 

0.0587 

2 

0 

2 

0.1892 

0.4586 

0.0533 

2 

0 

2 

0.7973 

1.2048 

0.1401 

8 

0 

8 

0.2162 

0.5304 

0.0617 

3 

0 

3 

0.1216 

0.3291 

0.0383 

1 

0 

1 

0.1081 

0.3906 

0.0454 

2 

0 

2 

0.1081 

0.3537 

0.0411 

2 

0 

2 

0.0270 

0.1633 

0.0190 

1 

0 

1 

0.3108 

0.5471 

0.0636 

2 

0 

2 

0.2297 

0.4547 

0.0529 

2 

0 

2 

1.0946 

1.5891 

0.1847 

7 

0 

7 

0.2027 

0.4048 

0.0471 

1 

0 

1 

0.0946 

0.2947 

0.0343 

1 

0 

1 

0.0541 

0.2277 

0.0265 

1 

0 

1 

0.2027 

0.4373 

0.0508 

2 

0 

2 

0.2297 

0.5114 

0.0595 

3 

0 

3 

0.1757 

0.5063 

0.0589 

2 

0 

2 

0.0676 

0.2527 

0.0294 

1 

0 

1 

0.4730 

0.6667 

0.0775 

3 

0 

3 

0.1081 

0.3906 

0.0454 

2 

0 

2 

0.1622 

0.4388 

0.0510 

2 

0 

2 

0.9459 

1.3838 

0.1609 

7 

0 

7 

1.3513 

1.1033 

0.1283 

5 

0 

5 

0.0541 

0.2815 

0.0327 

2 

0 

2 

0.9189 

1.2688 

0.1475 

5 

0 

5 

0.1757 

0.4785 

0.0556 

2 

0 

2 

0.5135 

0.6462 

0.0751 

2 

0 

2 

0.1757 

0.4174 

0.0485 

2 

0 

2 

0.1351 

0.3442 

0.0400 

1 

0 

1 

9.5135 

5.1398 

0.5975 

26 

0 

26 

10.4865 

1.0758 

0.1251 

14 

9 

5 

10.2703 

0.8648 

0.1005 

14 

9 

5 

11.0270 

1.8798 

0.2185 

19 

8 

11 

10.2027 

0.8273 

0.0962 

13 

7 

6 

10.2297 

0.6733 

0.0783 

13 

10 

3 

10.2027 

0.7211 

0.0838 

14 

10 

4 

10.1216 

0.6400 

0.0744 

14 

9 

5 

10.0270 

0.1633 

0.0190 

11 

10 

1 

10.4189 

0.8760 

0.1018 

14 

9 

5 

10.3784 

0.8712 

0.1013 

13 

9 

4 

11.3243 

2.4049 

0.2796 

25 

9 

16 

10.2973 

0.6565 

0.0763 

13 

10 

3 

10.1757 

0.6050 

0.0703 

13 

10 

3 

(Continued) 


I  • 

ria 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 

61 

62 

63 

64 

65 

66 

67 


4 

6 

5 

3 

7 

6 

4 

10 

9 

8 

11 

5 

6 

5 

3 

39 

6 

11 

13 

12 

1 

87 

16 


CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
LOWER  CRITERION  GROUP  (N-74) 
19R-EVALUATI0N  SECTION 
(Continued) 


S.E.  of 


Mean 

S.D. 

Mean 

Max. 

Min 

10.0405 

0.3070 

0.0357 

12 

9 

10.2432 

0.6582 

0.0765 

13 

9 

10.2162 

0.8321 

0.0967 

15 

9 

10.3108 

0.9352 

0.1087 

15 

10 

10.1081 

0.4553 

0.0529 

13 

10 

10.5676 

1.0609 

0.1233 

14 

7 

10.1216 

0.6611 

0.0769 

15 

9 

10.2973 

0.8231 

0.0957 

14 

10 

11.6621 

2.3366 

0.2716 

20 

10 

12.2973 

2.2618 

0.2629 

18 

9 

10.0946 

0.8137 

0.0946 

16 

8 

11.3919 

2.1696 

0.2522 

19 

8 

10.3108 

0.9641 

0.1121 

15 

10 

10.7973 

1.3546 

0.1575 

15 

9 

10.2027 

0.7018 

0.0816 

14 

9 

10.2297 

0.6313 

0.0734 

13 

10 

304.0640 

8.6142 

1.0014 

329 

290 

1.1351 

1.4835 

0.1724 

6 

0 

4.2432 

2.7339 

0.3178 

11 

0 

3.1621 

2.5537 

0.2969 

13 

0 

0.9595 

2.0703 

0.2407 

12 

0 

0.0135 

0.1162 

0.0135 

1 

0 

87.1889 

46.2204 

5.3730 

287 

0 

6.5135 

2.9759 

0.3459 

16 

0 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
UPPER  CRITERION  GROUP  (N-74) 

19S- JUSTIFICATION  SECTION 


No.  of 
Variable 

Mean 

S.D. 

S.E.  of 
Mean 

Max. 

Min. 

Range 

1 

0.4459 

0.7242 

0.0842 

3 

0 

3 

2 

0.7703 

1.3902 

0.1616 

8 

0 

8 

3 

1.9459 

1.9295 

0.2243 

8 

0 

8 

4 

0.1757 

0.4785 

0.0556 

3 

0 

3 

5 

0.4054 

0.7570 

0.0880 

3 

0 

3 

6 

0.1216 

0.3684 

0.0428 

2 

0 

2 

7 

0.2568 

0.7034 

0.0818 

4 

0 

4 

8 

0.0946 

0.4111 

0.0478 

3 

0 

3 

9 

0.8378 

1.0207 

0.1187 

4 

0 

4 

10 

1.1757 

1.3279 

0.1544 

5 

0 

5 

11 

2.0676 

1.9885 

0.2312 

7 

0 

7 

12 

0.5541 

0.8299 

0.0965 

3 

0 

3 

13 

0.2838 

0.5369 

0.0624 

2 

0 

2 

14 

0.0946 

0.3763 

0.0437 

2 

0 

2 

15 

0.7432 

1.1234 

0.1306 

5 

0 

5 

16 

0.6351 

0.8533 

0.0992 

4 

0 

4 

17 

0.4324 

0.8771 

0.1020 

5 

0 

5 

18 

0.5270 

0.7257 

0.0844 

3 

0 

3 

19 

1.0000 

1.0598 

0.1232 

5 

0 

5 

20 

0.5676 

0.8614 

0.1001 

3 

0 

3 

21 

0.5135 

0.7629 

0.0887 

4 

0 

4 

22 

1.5270 

1.7059 

0.1983 

8 

0 

8 

23 

2.5676 

2.2639 

0.2632 

14 

0 

14 

24 

0.1081 

0.3537 

0.0411 

2 

0 

2 

25 

2.2162 

2.2224 

0.2583 

10 

0 

10 

26 

0.3784 

0.6559 

0.0762 

3 

0 

3 

27 

1.1622 

1.0207 

0.1187 

4 

0 

4 

28 

0.3649 

0.6939 

0.0807 

3 

0 

3 

29 

0.4189 

0.6826 

0.0793 

3 

0 

3 

30 

22.3916 

14.7358 

1.7130 

91 

5 

86 

31 

11.0270 

1.7039 

0.1981 

18 

10 

8 

32 

11.3919 

2.3456 

0.2727 

23 

10 

13 

33 

13.9054 

3.8327 

0.4455 

28 

10 

18 

34 

10.2297 

0.6313 

0.0734 

13 

10 

3 

35 

10.8243 

1.5907 

0.1849 

18 

10 

8 

36 

10.2838 

0.8840 

0.1028 

15 

10 

5 

37 

10.4459 

1.2836 

0.1492 

18 

10 

8 

38 

10.1216 

0.4953 

0.0576 

13 

10 

3 

39 

11.5405 

2.0419 

0.2374 

17 

10 

7 

40 

12.4730 

2.7708 

0.3221 

20 

10 

10 

41 

13.8513 

3.6440 

0.4236 

23 

10 

13 

42 

11.1486 

1.8408 

0.2140 

18 

10 

8 

43 

10.5000 

1.1257 

0.1309 

16 

10 

6 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
UPPER  CRITERION  GROUP  (N=74) 

19S- JUSTIFICATION  SECTION 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

44 

10.1757 

0.7468 

0.0868 

15 

10 

5 

45 

11.5811 

2.4214 

0.2815 

20 

10 

10 

46 

11.0000 

1.5170 

0.1764 

17 

10 

7 

47 

10.7162 

1.6346 

0.1900 

19 

10 

9 

48 

10.9459 

1.3936 

0.1620 

15 

10 

5 

49 

11.9594 

2.1672 

0.2519 

18 

10 

8 

50 

10.9865 

1.5746 

0.1830 

17 

10 

7 

51 

10.6892 

1.3742 

0.1597 

20 

10 

10 

52 

13.1216 

3.4915 

0.4059 

28 

10 

18 

53 

15.5811 

4.6435 

0.5398 

39 

10 

29 

54 

10.3243 

1.0612 

0.1234 

16 

10 

6 

55 

14.0946 

4.1814 

0.4861 

28 

10 

18 

56 

10.5946 

1.0844 

0.1261 

15 

10 

5 

57 

12.4730 

2.3827 

0.2770 

19 

10 

9 

58 

10.5676 

1.0990 

0.1278 

15 

10 

5 

59 

10.7703 

1.2985 

0.1510 

15 

10 

5 

60 

333.3606 

27.1629 

3.1576 

449 

299 

150 

61 

6.8648 

10.7707 

1.2521 

91 

0 

91 

62 

9.9730 

8.5303 

0.9916 

59 

0 

59 

63 

7.1351 

6.7221 

0.7814 

44 

0 

44 

64 

0.0000 

0.0000 

0.0000 

0 

0 

0 

65 

0.0000 

0.0000 

0.0000 

0 

0 

0 

66 

201.4052 

146.3721 

17.0154 

896 

35 

861 

67 

11.6081 

4.8224 

0.5606 

27 

4 

23 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT’s) 
MIDDLE  CRITERION  GROUP  (N=74) 
19S-JUSTIFICATION  SECTION 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

i 

0.3919 

0.5690 

0.0661 

2 

0 

2 

2 

0.5135 

1.0369 

0.1205 

7 

0 

7 

3 

1.5540 

1.9525 

0.2270 

7 

0 

7 

4 

0.2162 

0.5040 

0.0586 

2 

0 

2 

5 

0.3784 

0.8227 

0.0956 

4 

0 

4 

6 

0.1622 

0.5740 

0.0667 

4 

0 

4 

7 

0.1486 

0.4588 

0.0533 

3 

0 

3 

8 

0.0676 

0.3021 

0.0351 

2 

0 

2 

9 

0.3919 

0.6583 

0.0765 

2 

0 

2 

10 

0.7973 

1.3846 

0.1610 

9 

0 

9 

11 

2.0000 

2.1390 

0.2487 

9 

0 

9 

12 

0.4595 

0.7797 

0.0906 

3 

0 

3 

13 

0.0946 

0.2947 

0.0343 

1 

0 

1 

14 

0.0811 

0.2748 

0.0319 

1 

0 

1 

15 

0.6081 

0.8730 

0.1015 

4 

0 

4 

16 

0.3919 

0.6153 

0.0715 

2 

0 

2 

17 

0.2568 

0.5250 

0.0610 

2 

0 

2 

18 

0.2838 

0.5618 

0.0653 

2 

0 

2 

19 

0.7432 

0.9801 

0.1139 

4 

0 

4 

20 

0.4730 

1.1494 

0.1336 

9 

0 

9 

21 

0.2703 

0.6259 

0.0728 

3 

0 

3 

22 

0.7432 

1.0076 

0.1171 

5 

0 

5 

23 

1.4865 

1.4358 

0.1669 

7 

0 

7 

24 

0.0405 

0.1986 

0.0231 

1 

0 

1 

25 

1.6892 

1.7354 

0.2017 

7 

0 

7 

26 

0.2162 

0.5304 

0.0617 

2 

0 

2 

27 

0.7838 

0.9107 

0.1059 

3 

0 

3 

28 

0.2703 

0.6036 

0.0702 

3 

0 

3 

29 

0.1757 

0.4174 

0.0485 

2 

0 

2 

30 

15.6890 

9.5657 

1.1120 

44 

1 

43 

31 

11.0270 

1.4709 

0.1710 

15 

10 

5 

32 

10.9594 

1.9120 

0.2223 

23 

10 

13 

33 

12.8919 

3.7106 

0.4314 

25 

10 

15 

34 

10.3919 

0.9482 

0.1102 

14 

10 

4 

35 

10.6216 

1.3619 

0.1583 

17 

10 

7 

36 

10.3784 

1.3314 

0.1548 

19 

10 

9 

37 

10.2297 

0.7503 

0.0872 

15 

10 

5 

38 

10.1216 

0.5722 

0.0665 

14 

10 

4 

39 

10.6216 

1.1312 

0.1315 

14 

10 

4 

40 

11.5946 

2.6062 

0.3030 

23 

10 

13 

41 

13.6081 

3.6067 

0.4193 

26 

10 

16 

42 

10.9324 

1.7621 

0.2048 

19 

10 

9 

43 

10.2162 

0.7266 

0.0845 

13 

10 

3 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
MIDDLE  CRITERION  GROUP  (N=74) 

19S- JUSTIFICATION  SECTION 
(Continued) 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

44 

10.1351 

0.5054 

0.0588 

13 

10 

3 

45 

11.3243 

1.8215 

0.2118 

17 

10 

7 

46 

10.5811 

1.1468 

0.1333 

16 

9 

7 

47 

10.4459 

0.9672 

0.1124 

14 

10 

4 

48 

10.5270 

1.0882 

0.1265 

14 

10 

4 

49 

11.3378 

1.8820 

0.2188 

17 

10 

7 

50 

10.8513 

1.8556 

0.2157 

23 

10 

13 

51 

10.4459 

1.1121 

0.1293 

16 

10 

6 

52 

11.2297 

1.7324 

0.2014 

17 

10 

7 

53 

13.1486 

3.2966 

0.3832 

25 

10 

15 

54 

10.1216 

0.5957 

0.0692 

13 

10 

3 

55 

12.9594 

3.4855 

0.4052 

27 

9 

18 

56 

10.4054 

1.0326 

0.1200 

15 

10 

5 

57 

11.4594 

1.7532 

0.2038 

17 

10 

7 

58 

10.4459 

1.0222 

0.1188 

15 

10 

5 

59 

10.3243 

0.8775 

0.1020 

15 

10 

5 

60 

319.3337 

17.1500 

1.9936 

371 

292 

79 

61 

3.5811 

2.7548 

0.3202 

13 

0 

13 

62 

6.5676 

4.4846 

0.5213 

21 

1 

20 

63 

5.5135 

4.7405 

0.5511 

18 

0 

18 

64 

0.0270 

0.2325 

0.0270 

2 

0 

2 

65 

0.0000 

0.0000 

0.0000 

0 

0 

0 

66 

146.5944 

124.8155 

14.5095 

820 

17 

803 

67 

9.1621 

4.3132 

0.5014 

19 

1 

18 
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CROSS  VALIDATION  SAMPLE  (AT's  and  BT's) 
LOWER  CRITERION  GROUP  (N=74) 

19S- JUSTIFICATION  SECTION 


No.  of 
Variable 

Mean 

S.D. 

1 

0.0541 

0.2815 

2 

0.0135 

0.1162 

3 

0.2027 

0.6406 

4 

0.0405 

0.1986 

5 

0.0405 

0.1986 

6 

0.0270 

0.2325 

7 

0.0135 

0.1162 

8 

0.0000 

0.0000 

9 

0.0270 

0.1633 

10 

0.1757 

0.5578 

11 

0.2838 

0.6728 

12 

0.1081 

0.4242 

13 

0.0270 

0.1633 

14 

0.0135 

0.1162 

15 

0.1892 

0.6553 

16 

0.0676 

0.3021 

17 

0.1081 

0.4553 

18 

0.0135 

0.1162 

19 

0.1081 

0.4242 

20 

0.0676 

0.3021 

21 

0.0270 

0.1633 

22 

0.1216 

0.3684 

23 

0.0946 

0.3380 

24 

0.0135 

0.1162 

25 

0.1351 

0.4480 

26 

0.0405 

0.1986 

27 

0.0946 

0.3763 

28 

0.0135 

0.1162 

29 

0.0405 

0.1986 

30 

2.1621 

3.5809 

31 

10.1216 

0.6611 

32 

10.0135 

0.1162 

33 

10.3919 

1.3830 

34 

10.0946 

0.5012 

35 

10.0676 

0.3445 

36 

10.0676 

0.5812 

37 

10.0270 

0.2325 

38 

10.0000 

0.0000 

39 

10.0540 

0.3661 

40 

10.3513 

1.2212 

41 

10.5946 

1.5432 

42 

10.2297 

0.8528 

43 

10.0676 

0.4165 

S.E.  of 


Mean 

Max. 

Min. 

Range 

0.0327 

2 

0 

2 

0.0135 

1 

0 

1 

0.0745 

4 

0 

4 

0.0231 

1 

0 

1 

0.0231 

1 

0 

1 

0.0270 

2 

0 

2 

0.0135 

1 

0 

1 

0.0000 

0 

0 

0 

0.0190 

1 

0 

1 

0.0648 

3 

0 

3 

0.0782 

3 

0 

3 

0.0493 

3 

0 

3 

0.0190 

1 

0 

1 

0.0135 

1 

0 

1 

0.0762 

5 

0 

5 

0.0351 

2 

0 

2 

0.0529 

3 

0 

3 

0.0135 

1 

0 

1 

0.0493 

2 

0 

2 

0.0351 

2 

0 

2 

0.0190 

1 

0 

1 

0.0428 

2 

0 

2 

0.0393 

2 

0 

2 

0.0135 

1 

0 

1 

0.0521 

2 

0 

2 

0.0231 

1 

0 

1 

0.0437 

2 

0 

2 

0.0135 

1 

0 

1 

0.0231 

1 

0 

1 

0.4163 

16 

0 

16 

0.0769 

15 

10 

5 

0.0135 

11 

10 

1 

0.1608 

20 

10 

10 

0.0583 

13 

10 

3 

0.0400 

12 

10 

2 

0.0676 

15 

10 

5 

0.0270 

12 

10 

2 

0.0000 

10 

10 

0 

0.0426 

13 

10 

3 

0.1420 

16 

9 

7 

0.1794 

17 

9 

8 

0.0991 

15 

10 

5 

0.0484 

13 

10 

3 

(Continued) 


163 


CROSS  VALIDATION  SAMPLE  (AT’s  and  BT’s) 
LOWER  CRITERION  GROUP  (N-74) 
19S-JUSTIFICATION  SECTION 
(Continued) 


No.  of 
Variable 

Mean 

S.D. 

44 

10.0405 

0.3487 

45 

10.4730 

1.5280 

46 

10.1081 

0.5381 

47 

10.2162 

0.9547 

48 

10.0270 

0.2325 

49 

10.2703 

1.0765 

50 

10.1351 

0.6265 

51 

10.0540 

0.3265 

52 

10.2162 

0.6677 

53 

10.1621 

0.5974 

54 

9.9730 

0.2325 

55 

10.2297 

0.7503 

56 

10.0811 

0.3971 

57 

10.1486 

0.6123 

58 

10.0405 

0.3487 

59 

10.0676 

0.3822 

60 

294.1848 

6.8917 

61 

0.6486 

1.2212 

62 

0.9595 

1.8464 

63 

0.5135 

1.0500 

64 

0.0270 

0.1633 

65 

0.0135 

0.1162 

66 

17.6754 

27.2311 

67 

1.6081 

2.4372 

S.E.  of 


Mean 

Max. 

Min. 

Range 

0.0405 

13 

10 

3 

0.1776 

21 

10 

11 

0.0625 

14 

10 

4 

0.1110 

17 

10 

7 

0.0270 

12 

10 

2 

0.1251 

16 

10 

6 

0.0728 

14 

10 

4 

0.0380 

12 

10 

2 

0.0776 

13 

10 

3 

0.0694 

13 

10 

3 

0.0270 

10 

8 

2 

0.0872 

14 

10 

4 

0.0462 

12 

10 

2 

0.0712 

14 

10 

4 

0.0405 

13 

10 

3 

0.0444 

13 

10 

3 

0.8011 

324 

287 

37 

0.1420 

7 

0 

7 

0.2146 

10 

0 

10 

0.1221 

5 

0 

5 

0.0190 

1 

0 

1 

0.0135 

1 

0 

1 

3.1656 

112 

0 

112 

0.2833 

9 

0 

9 

•  I 

ia 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 


GENERALIZATION  SAMPLE  (CS's  and  RM's) 
UPPER  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 


S.E.  of 


Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

0.2973 

0.5163 

0.0600 

2 

0 

2 

0.3378 

0.8802 

0.1023 

4 

0 

4 

0.9054 

1.4634 

0.1701 

7 

0 

7 

0.1851 

0.3819 

0.0444 

2 

0 

2 

0.2703 

0.6884 

0.0800 

4 

0 

4 

0.0676 

0.2527 

0.0294 

1 

0 

1 

0.1081 

0.3126 

0.0363 

1 

0 

1 

0.0135 

0.1162 

0.0135 

1 

0 

1 

0.3243 

0.7043 

0.0819 

4 

0 

4 

0.2703 

0.6682 

0.0777 

3 

0 

3 

1.0405 

1.7629 

0.2049 

10 

0 

10 

0.2973 

0.6353 

0.0738 

3 

0 

3 

0.0405 

0.1986 

0.0231 

1 

0 

1 

0.0541 

0.2815 

0.0327 

2 

0 

2 

0.1757 

0.4490 

0.0522 

2 

0 

2 

0.2162 

0.5040 

0.0586 

2 

0 

2 

0.2027 

0.5229 

0.0608 

3 

0 

3 

0.3919 

0.8246 

0.0959 

5 

0 

5 

0.2297 

0.4839 

0.0563 

2 

0 

2 

0.1351 

0.4776 

0.0555 

3 

0 

3 

0.0811 

0.3610 

0.0420 

2 

0 

2 

0.7703 

1.0142 

0.1179 

4 

0 

4 

1.5000 

1.5011 

0.1745 

7 

0 

7 

0.2703 

0.4768 

0.0554 

2 

0 

2 

0.9459 

1.2038 

0.1399 

5 

0 

5 

0.1622 

0.4388 

0.0510 

2 

0 

2 

0.3514 

0.6288 

0.0731 

3 

0 

3 

0.2838 

0.5857 

0.0681 

3 

0 

3 

0.2162 

0.4760 

0.0553 

2 

0 

2 

10.0946 

9.6927 

1.1268 

63 

0 

63 

10.6621 

1.2306 

0.1431 

15 

10 

5 

10.6892 

1.7511 

0.2036 

19 

10 

9 

11.7973 

3.0965 

0.3600 

29 

10 

19 

10.2973 

0.9025 

0.1049 

15 

10 

5 

10.5540 

1.3461 

0.1565 

17 

10 

7 

10.1486 

0.5657 

0.0658 

13 

10 

3 

10.1892 

0.5893 

0.0685 

13 

10 

3 

10.0270 

0.2325 

0.0270 

12 

10 

2 

10.6216 

1.3619 

0.1583 

18 

10 

8 

10.5946 

1.5253 

0.1773 

18 

10 

8 

11.7703 

3.3085 

0.3846 

32 

10 

22 

10.5811 

1.2497 

0.1453 

16 

10 

6 

10.0676 

0.3822 

0.0444 

13 

10 

3 

(Continued) 
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GENERALIZATION  SAMPLE  (CS's  and  RM's) 
UPPER  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 
(Continued) 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

44 

10.1081 

0.5630 

0.0654 

14 

10 

4 

45 

10.3784 

1.0942 

0.1272 

16 

10 

6 

46 

10.3649 

0.9872 

0.1148 

15 

10 

5 

47 

10.3513 

0.8980 

0.1044 

14 

10 

4 

48 

10.8378 

1.6719 

0.1944 

19 

10 

9 

49 

10.4594 

1.1003 

0.1279 

16 

10 

6 

50 

10.2162 

0.7811 

0.0908 

15 

10 

5 

51 

10.1081 

0.5120 

0.0595 

13 

10 

3 

52 

11.5946 

2.0668 

0.2403 

18 

10 

8 

53 

13.5405 

3.7968 

0.4414 

31 

10 

21 

54 

10.8108 

1.4304 

0.1663 

16 

10 

6 

55 

11.6892 

2.2141 

0.2574 

20 

10 

10 

56 

10.3108 

0.9641 

0.1121 

16 

10 

6 

57 

10.6757 

1.2939 

0.1504 

17 

10 

7 

58 

10.5000 

1.1849 

0.1377 

17 

10 

7 

59 

10.4865 

1.1133 

0.1294 

15 

10 

5 

60 

310.5364 

21.1002 

2.4528 

445 

290 

155 

61 

2.9730 

4.0102 

0.4662 

32 

0 

32 

62 

4.4189 

4.2716 

0.4966 

19 

0 

19 

63 

2.6892 

3.4240 

0.3980 

16 

0 

16 

64 

0.0135 

0.1162 

0.0135 

1 

0 

1 

65 

0.0000 

0.0000 

0.0000 

0 

0 

0 

66 

98.1754 

83.3072 

9.6843 

530 

0 

530 

67 

6.3378 

4.7637 

0.5538 

22 

0 

22 
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GENERALIZATION  SAMPLE  (OS's  and  RM's) 
MIDDLE  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

i 

0.4595 

0.6862 

0.798 

3 

0 

3 

2 

0.3514 

0.8012 

0.0931 

4 

0 

4 

3 

0.8649 

1.1387 

0.1324 

5 

0 

5 

4 

0.0946 

0.2947 

0.0343 

1 

0 

1 

5 

0.1622 

0.4064 

0.0472 

2 

0 

2 

6 

0.1486 

0.5150 

0.0599 

3 

0 

3 

7 

0.2027 

0.5229 

0.0608 

2 

0 

2 

8 

0.0405 

0.1986 

0.0231 

1 

0 

1 

9 

0.2297 

0.4547 

0.0529 

2 

0 

2 

10 

0.1622 

0.4973 

0.0578 

3 

0 

3 

11 

0.6351 

1.2002 

0.1395 

5 

0 

5 

12 

0.2027 

0.4676 

0.0544 

2 

0 

2 

13 

0.0135 

0.1162 

0.0135 

1 

0 

1 

14 

0.0541 

0.2277 

0.0265 

1 

0 

1 

15 

0.1216 

0.3684 

0.0428 

2 

0 

2 

16 

0.1892 

0.3943 

0.0458 

1 

0 

1 

17 

0.0946 

0.3380 

0.0393 

2 

0 

2 

18 

0.2838 

0.5369 

0.0624 

2 

0 

2 

19 

0.4189 

0.7216 

0.0839 

3 

0 

3 

20 

0.1216 

0.3684 

0.0428 

2 

0 

2 

21 

0.2027 

0.4676 

0.0544 

2 

0 

2 

22 

0.6892 

1.1339 

0.1318 

6 

0 

6 

23 

1.3243 

1.4440 

0.1679 

8 

0 

8 

24 

0.1622 

0.4064 

0.0472 

2 

0 

2 

25 

0.9865 

1.2219 

0.1420 

5 

0 

5 

26 

0.0541 

0.2277 

0.0265 

1 

0 

1 

27 

0.1622 

0.4064 

0.0472 

2 

0 

2 

28 

0.1757 

0.4490 

0.0522 

2 

0 

2 

29 

0.1216 

0.3291 

0.0383 

1 

0 

1 

30 

8.7297 

5.5967 

0.6506 

26 

0 

26 

31 

11.0135 

1.4571 

0.1694 

15 

10 

5 

32 

10.5811 

1.5439 

0.1795 

18 

7 

11 

33 

11.5946 

2.1891 

0.2545 

19 

10 

9 

34 

10.2162 

0.7075 

0.0822 

13 

10 

3 

35 

10.2838 

0.7498 

0,0872 

13 

10 

3 

36 

10.2973 

0.9613 

0.1117 

14 

10 

4 

37 

10.3513 

1.0782 

0.1253 

16 

9 

7 

38 

10.0811 

0.3971 

0.0462 

12 

10 

2 

39 

10.3648 

0.8205 

0.0954 

13 

10 

3 

40 

10.2973 

1.0301 

0.1197 

15 

9 

6 

41 

10.8649 

1.5644 

0.1819 

16 

10 

6 

42 

10.2703 

0.6682 

0.0777 

13 

10 

3 

43 

10.0270 

0.2325 

0.0270 

12 

10 

2 
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GENERALIZATION  SAMPLE  (CS's  and  RM's) 
MIDDLE  CRITERION  GROUP  (N«74) 

19R- EVALUATION  SECTION 
(Continued) 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

44 

10.1081 

0.4553 

0.0529 

12 

10 

2 

45 

10.2838 

0.8524 

0.0991 

14 

10 

4 

46 

10.3513 

0.7840 

0.0911 

13 

10 

3 

47 

10.1351 

0.4776 

0.0555 

12 

10 

2 

48 

10.4865 

0.9826 

0.1142 

14 

10 

4 

49 

10.6216 

1.1786 

0.1370 

15 

9 

6 

50 

10.2027 

0.6617 

0.0769 

14 

10 

4 

51 

10.2567 

0.6424 

0.0747 

13 

10 

3 

52 

11.4324 

2.3059 

0.2681 

22 

10 

12 

53 

12.7162 

2.9633 

0.3445 

26 

10 

16 

54 

10.4865 

1.2191 

0.1417 

16 

10 

6 

55 

11.7432 

2.2638 

0.2632 

20 

10 

10 

56 

10.0811 

0.3971 

0.0462 

13 

10 

3 

57 

10.1892 

0.5893 

0.0685 

13 

9 

4 

58 

10.2567 

0.6837 

0.0795 

13 

10 

3 

59 

10.2567 

0.7034 

0.0818 

13 

10 

3 

60 

305.8467 

10.0993 

1.1740 

338 

290 

48 

61 

1.8513 

1.7570 

0.2043 

7 

0 

7 

62 

3.8784 

3.0608 

0.3558 

11 

0 

11 

63 

2.7703 

2.5831 

0.3003 

11 

0 

11 

64 

0.2297 

0.8687 

0.1010 

6 

0 

6 

65 

0.0000 

0.0000 

0.0000 

0 

0 

0 

66 

84.5132 

53.0552 

6.1675 

293 

0 

293 

67 

5.8648 

3.3365 

0.3879 

14 

0 

14 
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GENERALIZATION  SAMPLE  (OS's  and  RM's) 
LOWER  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

i 

0.2297 

0.4547 

0.0529 

2 

0 

2 

2 

0.3649 

0.6532 

0.0759 

3 

0 

3 

3 

0.8378 

1.1229 

0.1305 

6 

0 

6 

4 

0.1486 

0.4279 

0.0497 

2 

0 

2 

5 

0.2432 

0.4320 

0.0502 

1 

0 

1 

6 

0.0270 

0.1633 

0.0190 

1 

0 

1 

7 

0.1622 

0.4388 

0.0510 

2 

0 

2 

8 

0.0135 

0.1162 

0.0135 

1 

0 

1 

9 

0.2703 

0.5311 

0.0617 

2 

0 

2 

10 

0.4324 

0.7038 

0.0818 

3 

0 

3 

11 

0.8378 

1.1229 

0.1305 

5 

0 

5 

12 

0.1892 

0.4277 

0.0497 

2 

0 

2 

13 

0.0000 

0.0000 

0.0000 

0 

0 

0 

14 

0.1216 

0.4038 

0.0469 

2 

0 

2 

15 

0.1757 

0.4785 

0.0556 

3 

0 

3 

16 

0.1351 

0.3819 

0.0444 

2 

0 

2 

17 

0.0946 

0.3380 

0.0393 

2 

0 

2 

18 

0.2162 

0.4463 

0.0519 

2 

0 

2 

19 

0.4189 

0.7586 

0.0882 

3 

0 

3 

20 

0.1757 

0.4785 

0.0556 

2 

0 

2 

21 

0.1351 

0.3442 

0.0400 

1 

0 

1 

22 

0.6486 

0.9280 

0.1079 

4 

0 

4 

23 

1.6621 

1.5374 

0.1787 

7 

0 

7 

24 

0.1081 

0.3537 

0.0411 

2 

0 

2 

25 

0.9865 

1.1878 

0.1381 

4 

0 

4 

26 

0.1216 

0.3684 

0.0428 

2 

0 

2 

27 

0.3784 

0.6127 

0.0712 

3 

0 

3 

28 

0.1622 

0.4388 

0.0510 

2 

0 

2 

29 

0.1351 

0.3442 

0.0400 

1 

0 

1 

30 

9.4324 

5.0965 

0.5925 

29 

0 

29 

31 

10.4189 

0.9364 

0.1089 

13 

9 

4 

32 

10.5676 

1.0479 

0.1218 

14 

10 

4 

33 

11.0946 

2.1592 

0.2510 

23 

7 

16 

34 

10.2838 

0.8196 

0.0953 

14 

10 

4 

35 

10.4730 

0.8793 

0.1022 

13 

10 

3 

36 

10.0676 

0.4165 

0.0484 

13 

10 

3 

37 

10.2838 

0.8027 

0.0933 

14 

10 

4 

38 

10.0270 

0.2325 

0.0270 

12 

10 

2 

39 

10.3919 

0.9189 

0.1068 

14 

9 

5 

40 

10.6081 

1.1915 

0.1385 

15 

9 

6 

41 

11.1486 

2.0385 

0.2370 

19 

8 

11 

42 

10.3108 

0.8264 

0.0961 

13 

9 

4 

43 

10.0000 

0.0000 

0.0000 

10 

10 

0 
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GENERALIZATION  SAMPLE  (CS's  and  RM's) 
LOWER  CRITERION  GROUP  (N=74) 

19R- EVALUATION  SECTION 
(Continued) 


No.  of 


S.E.  of 


iable 

Mean 

S.D. 

Mean 

Max. 

Min . 

Range 

44 

10.0811 

0.5675 

0.0660 

13 

8 

5 

45 

10.3784 

1.0689 

0.1243 

17 

10 

7 

46 

10.1757 

0.6487 

0.0754 

14 

9 

5 

47 

10.1486 

0.5410 

0.0629 

13 

10 

3 

48 

10.3378 

0.7635 

0.0888 

13 

10 

3 

49 

10.7432 

1.4340 

0.1667 

17 

10 

7 

50 

10.2703 

0.7816 

0.0909 

14 

10 

4 

51 

10.1757 

0.4785 

0.0556 

12 

10 

2 

52 

11.1351 

1.7698 

0.2057 

18 

9 

9 

53 

13.1757 

3.3407 

0.3884 

22 

8 

14 

54 

10.2567 

1.0476 

0.1218 

16 

8 

8 

55 

11.3378 

2.0155 

0.2343 

18 

8 

10 

56 

10.2027 

0.6617 

0.0769 

13 

10 

3 

57 

10.7567 

1.4026 

0.1631 

17 

9 

8 

58 

10.2162 

0.6677 

0.0776 

14 

10 

4 

59 

10.2432 

0.6582 

0.0765 

13 

10 

3 

60 

305.5764 

9.0569 

1.0528 

333 

290 

43 

61 

1.5540 

1.5362 

0.1786 

7 

0 

7 

62 

4.2838 

2.6767 

0.3112 

16 

0 

16 

63 

2.8513 

2.4700 

0.2871 

9 

0 

9 

64 

0.7297 

1.5641 

0.1818 

9 

0 

9 

65 

0.0135 

0.1162 

0.0135 

1 

0 

1 

66 

100.2835 

54.5660 

6.3432 

353 

0 

353 

67 

6.3648 

2.8020 

0.3257 

14 

0 

14 

170 


GENERALIZATION  SAMPLE  (OS's  and  RM’s) 
UPPER  CRITERION  GROUP  (N=74) 

19S- JUSTIFICATION  SECTION 


No.  of 
Variable 

Mean 

S .  D. 

S.E.  of 
Mean 

Max. 

Min. 

Range 

1 

0.5135 

0.8951 

0.1041 

4 

0 

4 

2 

0.8243 

1.3381 

0.1556 

8 

0 

8 

3 

2.0811 

1.9846 

0.2307 

9 

0 

9 

4 

0.2568 

0.6633 

0.0771 

3 

0 

3 

5 

0.6081 

1.0831 

0.1259 

6 

0 

6 

6 

0.1622 

0.5241 

0.0609 

3 

0 

3 

7 

0.1757 

0.3831 

0.0445 

1 

0 

1 

8 

0.0946 

0.2947 

0.0343 

1 

0 

1 

9 

0.9189 

1.1196 

0.1302 

5 

0 

5 

10 

1.2567 

1.8061 

0.2100 

9 

0 

9 

11 

2.4865 

2.7411 

0.3186 

14 

0 

14 

12 

0.8784 

0.9209 

0.1070 

4 

0 

4 

13 

0.1486 

0.3946 

0.0459 

2 

0 

2 

14 

0.0811 

0.2748 

0.0319 

1 

0 

1 

15 

0.8784 

0.8101 

0.0942 

4 

0 

4 

16 

0.8649 

0.8809 

0.1024 

4 

0 

4 

17 

0.3649 

0.6739 

0.0783 

2 

0 

2 

18 

0.5946 

0.8263 

0.0961 

3 

0 

3 

19 

1.4189 

1.3650 

0.1587 

6 

0 

6 

20 

0.6351 

0.8848 

0.1029 

4 

0 

4 

21 

0.2838 

0.5369 

0.0624 

2 

0 

2 

22 

1.4054 

1.5955 

0.1855 

6 

0 

6 

23 

2.5270 

2.0555 

0.2390 

8 

0 

8 

24 

0.0135 

0.1162 

0.0135 

1 

0 

1 

25 

2.6757 

2.4220 

0.2815 

10 

0 

10 

26 

0.4595 

0.6660 

0.0774 

3 

0 

3 

27 

1.1351 

1.1387 

0.1324 

4 

0 

4 

28 

0.2973 

0.5669 

0.0659 

2 

0 

2 

29 

0.5135 

0.7070 

0.0822 

3 

0 

3 

30 

24.5673 

15.3846 

1.7884 

75 

2 

73 

31 

11.2027 

2.0804 

0.2418 

20 

10 

10 

32 

11.4189 

2.3815 

0.2768 

24 

10 

14 

33 

14.0540 

4.0776 

0.4740 

27 

10 

17 

34 

10.4594 

1.1958 

0.1390 

16 

10 

6 

35 

11.0811 

1.8635 

0.2166 

20 

10 

10 

36 

10.3378 

1.0634 

0.1236 

16 

10 

6 

37 

10.3108 

0.7389 

0.0859 

13 

10 

3 

38 

10.1621 

0.5496 

0.0639 

13 

10 

3 

39 

11.7567 

2.2071 

0.2566 

20 

10 

10 

40 

12.5811 

3.5654 

0.4145 

27 

9 

18 

41 

14.6621 

4.2500 

0.4940 

29 

10 

19 

42 

11.7027 

1.9705 

0.2291 

20 

10 

10 

43 

10.2703 

0.7457 

0.0867 

13 

10 

3 

(Continued) 
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GENERALIZATION  SAMPLE  (CS's  and  RM's) 
UPPER  CRITERION  GROUP  (N-74) 

19S- JUSTIFICATION  SECTION 
(Continued) 


No.  of 

Variable  Mean  S.D. 


44 

10.1621 

0.5974 

45 

11.9594 

1.7629 

46 

11.3648 

1.4858 

47 

10.5811 

1.1104 

48 

11.0946 

1.6316 

49 

12.6081 

2.6266 

50 

11.1757 

1.6163 

51 

10.3784 

0.7711 

52 

12.7838 

3.3445 

53 

15.8647 

4.6855 

54 

10.0405 

0.3487 

55 

14.5811 

3.8924 

56 

10.7432 

1.2612 

57 

12.2838 

2.3671 

58 

10.5000 

1.0102 

59 

11.0135 

1.4665 

60 

337.1443 

28.4735 

61 

6.0946 

4.5183 

62 

10.5811 

6.9420 

63 

7.8243 

6.8012 

64 

0.0676 

0.4778 

65 

0.0000 

0.0000 

66 

228.2968 

155.7997 

67 

12.4730 

4.6852 

S.E.  of 


Mean 

Max. 

Min. 

Range 

0.0694 

13 

10 

3 

0.2049 

17 

10 

7 

0.1727 

16 

10 

6 

0.1291 

14 

10 

4 

0.1897 

16 

10 

6 

0.3053 

22 

10 

12 

0.1879 

16 

10 

6 

0.0896 

13 

10 

3 

0.3888 

23 

10 

13 

0.5447 

30 

10 

20 

0.0405 

13 

10 

3 

0.4525 

23 

10 

13 

0.1466 

16 

10 

6 

0.2752 

18 

10 

8 

0.1174 

14 

10 

4 

0.1705 

15 

10 

5 

3.3100 

423 

294 

129 

0.5252 

20 

0 

20 

0.8070 

32 

1 

31 

0.7906 

31 

0 

31 

0.0555 

4 

0 

4 

0.0000 

0 

0 

0 

18.1113 

881 

17 

864 

0.5446 

22 

1 

21 

.  \ 

ial 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 


GENERALIZATION  SAMPLE  (CS's  and  RM's) 
MIDDLE  CRITERION  GROUP  (N=74) 
19S-JUSTIFICATI0N  SECTION 


S.E.  of 


Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

0.3108 

0.6395 

0.0743 

2 

0 

2 

0.3919 

0.8246 

0.0959 

3 

0 

3 

1.0946 

1.6232 

0.1887 

7 

0 

7 

0.2027 

0.6617 

0.0769 

4 

0 

4 

0.4324 

0.7417 

0.0862 

3 

0 

3 

0.1081 

0.5381 

0.0625 

4 

0 

4 

0.1622 

0.4690 

0.0545 

3 

0 

3 

0.0405 

0.2585 

0.0301 

2 

0 

2 

0.4865 

0.7806 

0.0907 

3 

0 

3 

0.4324 

0.8289 

0.0964 

4 

0 

4 

1.5676 

1.7910 

0.2082 

7 

0 

7 

0.4054 

0.7199 

0.0837 

3 

0 

3 

0.1892 

0.5893 

0.0685 

3 

0 

3 

0.0811 

0.3208 

0.0373 

2 

0 

2 

0.7973 

0.9506 

0.1105 

4 

0 

4 

0.4730 

0.6458 

0.0751 

3 

0 

3 

0.1892 

0.4875 

0.0567 

3 

0 

3 

0.2297 

0.5376 

0.0625 

3 

0 

3 

0.8919 

0.9871 

0.1148 

4 

0 

4 

0.3649 

0.7508 

0.0873 

3 

0 

3 

0.2703 

0.5563 

0.0647 

2 

0 

2 

0.5811 

0.9509 

0.1105 

5 

0 

5 

2.0270 

1.8870 

0.2194 

11 

0 

11 

0.0000 

0.0000 

0.0000 

0 

0 

0 

1.4324 

1.6393 

0.1906 

7 

0 

7 

0.2162 

0.5304 

0.0617 

3 

0 

3 

0.5405 

0.7251 

0.0843 

3 

0 

3 

0.1622 

0.3711 

0.0431 

1 

0 

1 

0.2838 

0.5369 

0.0624 

3 

0 

3 

14.3648 

9.8961 

1.1504 

49 

0 

49 

10.7567 

1.5860 

0.1844 

16 

10 

6 

10.6621 

1.4456 

0.1680 

16 

9 

7 

12.1351 

3.2449 

0.3772 

27 

10 

17 

10.3243 

1.0351 

0.1203 

16 

10 

6 

10.8108 

1.5141 

0.1760 

16 

10 

6 

10.1892 

0.9017 

0.1048 

16 

10 

6 

10.2838 

0.9583 

0.1114 

17 

10 

7 

10.0541 

0.3265 

0.0380 

12 

10 

2 

10.8513 

1.4304 

0.1663 

15 

10 

5 

10.8513 

1.6021 

0.1862 

16 

10 

6 

12.7567 

3.0780 

0.3578 

21 

10 

11 

10.7973 

1.5168 

0.1763 

16 

10 

6 

10.2973 

0.9754 

0.1134 

16 

10 

6 
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GENERALIZATION  SAMPLE  (OS's  and  RM's) 
MIDDLE  CRITERION  GROUP  (N=74) 

19S- JUSTIFICATION  SECTION 
(Continued) 


No.  of  S.E.  of 


Variable 

Mean 

S.D. 

Mean 

Max. 

Min. 

Range 

44 

10.1351 

0.5318 

0.0618 

13 

10 

3 

45 

11.8108 

2.3035 

0.2678 

21 

10 

11 

46 

10.6892 

0.9782 

0.1137 

14 

10 

4 

47 

10.3513 

0.9131 

0.1061 

15 

10 

5 

48 

10.3919 

1.0313 

0.1199 

16 

10 

6 

49 

11.5000 

1.8672 

0.2171 

19 

10 

9 

50 

10.5135 

1.1010 

0.1280 

15 

10 

5 

51 

10.3784 

0.8392 

0.0975 

14 

10 

4 

52 

11.0811 

1.7102 

0.1988 

19 

10 

9 

53 

14.2297 

4.1266 

0.4797 

33 

10 

23 

54 

10.0000 

0.0000 

0.0000 

10 

10 

0 

55 

12.4324 

2.9381 

0.3415 

23 

10 

13 

56 

10.3649 

0.9447 

0.1098 

15 

10 

5 

57 

11.0540 

1.6124 

0.1874 

16 

10 

6 

58 

10.2567 

0.6424 

0.0747 

13 

10 

3 

59 

10.5676 

1.1236 

0.1306 

16 

10 

6 

60 

316.5225 

18.2112 

2.1170 

374 

290 

84 

61 

3.1216 

2.7843 

0.3237 

14 

0 

14 

62 

6.0000 

4.7571 

0.5530 

25 

0 

25 

63 

5.2027 

4.5510 

0.5290 

19 

0 

19 

64 

0.0405 

0.1986 

0.0231 

1 

0 

1 

65 

0.0000 

0.0000 

0.0000 

0 

0 

0 

66 

133.0268 

96.9929 

11.2752 

442 

0 

442 

67 

8.4730 

4.2819 

0.4978 

18 

0 

18 

* 
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GENERALIZATION  SAMPLE  (CS's  and  RM's) 
LOWER  CRITERION  GROUP  (N»74) 

19S- JUSTIFICATION  SECTION 


No.  of  S.E.  of 


Variable 

Mean 

S .  D. 

Mean 

Max. 

Min. 

Range 

i 

0.0135 

0.1162 

0.0135 

i 

0 

1 

2 

0.0676 

0.3021 

0.0351 

2 

0 

2 

3 

0.1351 

0.5318 

0.0618 

4 

0 

4 

4 

0.0405 

0.1986 

0.0231 

1 

0 

1 

5 

0.0541 

0.2815 

0.0327 

2 

0 

2 

6 

0.0135 

0.1162 

0.0135 

1 

0 

1 

7 

0.0676 

0.3445 

0.0400 

2 

0 

2 

8 

0.0270 

0.1633 

0.0190 

1 

0 

1 

9 

0.0405 

0.2585 

0.0301 

2 

0 

2 

10 

0.2432 

0.6985 

0.0812 

4 

0 

4 

11 

0.3514 

0.8182 

0.0951 

4 

0 

4 

12 

0.2297 

0.6092 

0.0708 

3 

0 

3 

13 

0.0135 

0.1162 

0.0135 

1 

0 

1 

14 

0.0270 

0.1633 

0.0190 

1 

0 

1 

15 

0.3243 

0.9524 

0.1107 

5 

0 

5 

16 

0.0811 

0.4898 

0.0569 

4 

0 

4 

17 

0.0541 

0.2815 

0.0327 

2 

0 

2 

18 

0.0135 

0.1162 

0.0135 

1 

0 

1 

19 

0.1216 

0.4038 

0.0469 

2 

0 

2 

20 

0.0541 

0.2277 

0.0265 

1 

0 

1 

21 

0.0676 

0.2527 

0.0294 

1 

0 

1 

22 

0.1216 

0.4668 

0.0543 

3 

0 

3 

23 

0.1757 

0.5819 

0.0676 

4 

0 

4 

24 

0.0000 

0.0000 

0.0000 

0 

0 

0 

25 

0.1216 

0.4668 

0.0543 

3 

0 

3 

26 

0.0541 

0.2277 

0.0265 

1 

0 

1 

27 

0.2027 

0.5729 

0.0666 

2 

0 

2 

28 

0.0135 

0.1162 

0.0135 

1 

0 

1 

29 

0.0405 

0.1986 

0.0231 

1 

0 

1 

30 

2.7703 

4.0798 

0.4743 

21 

0 

21 

31 

10.0270 

0.2325 

0.0270 

12 

10 

2 

32 

10.1081 

0.4845 

0.0563 

13 

10 

3 

33 

10.2567 

1.0476 

0.1218 

18 

10 

8 

34 

10.0405 

0.1986 

0.0231 

11 

10 

1 

35 

10.0946 

0.4731 

0.0550 

13 

10 

3 

36 

10.0270 

0.2325 

0.0270 

12 

10 

2 

37 

10.1351 

0.6688 

0.0777 

14 

10 

4 

38 

10.0540 

0.3661 

0.0426 

13 

10 

3 

39 

10.0676 

0.4165 

0.0484 

13 

10 

3 

40 

10.4189 

1.4046 

0.1633 

18 

9 

9 

41 

10.8513 

1.9492 

0.2266 

19 

10 

9 

42 

10.5135 

1.4069 

0.1635 

17 

10 

7 

43 

10.0135 

0.1162 

0.0135 

11 

10 

1 

(Continued) 


:ia 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 

61 

62 

63 

64 

65 

66 

67 


2 

12 

7 

4 

2 

4 

3 

2 

5 

6 

0 

5 

2 

5 

2 

2 

34 

4 

12 

9 

1 

0 

72 

10 


GENERAL IZATION  SAMPLE  (OS's  and  RM's) 
LOWER  CRITERION  GROUP  (N=74) 

19S- JUSTIFICATION  SECTION 
(Continued) 


S.E.  of 


Mean 

S.D. 

Mean 

Max. 

Min 

10.0405 

0.2585 

0.0301 

12 

10 

10.6892 

2.1062 

0.2448 

21 

9 

10.1351 

0.8492 

0.0987 

17 

10 

10.1216 

0.6183 

0.0719 

14 

10 

10.0270 

0.2325 

0.0270 

12 

10 

10.2162 

0.7266 

0.0845 

14 

10 

10.1351 

0.5811 

0.0675 

13 

10 

10.0946 

0.3763 

0.0437 

12 

10 

10.1892 

0.7706 

0.0896 

15 

10 

10.2703 

0.9260 

0.1076 

16 

10 

10.0000 

0.0000 

0.0000 

10 

10 

10.2567 

0.9225 

0.1072 

15 

10 

10.0676 

0.3021 

0.0351 

12 

10 

10.4054 

1.1811 

0.1373 

15 

10 

10.0270 

0.2325 

0.0270 

12 

10 

10.0676 

0.3445 

0.0400 

12 

10 

295.3472 

7.4695 

0.8683 

323 

289 

0.6216 

0.9751 

0.1133 

4 

0 

1.4189 

2.2514 

0.2617 

12 

0 

0.6892 

1.5432 

0.1794 

9 

0 

0.0405 

0.1986 

0.0231 

1 

0 

0.0000 

0.0000 

0.0000 

0 

0 

27.4051 

40.0231 

4.6526 

172 

0 

1.8648 

2.5660 

0.2983 

10 

0 

176 


177 


APPENDIX  C 

CORRELATIONS  AMONG  THE  67  VARIABLES* 


CROSS  VALIDATION  SAMPLE  -  AT's  AND  BT's  COMBINED  (N-222) 


EVALUATION  SECTION  (19R) 

JUSTIFICATION  SECTION  (19S) 

EVALUATION  SECTION  (19R)  VS. 
JUSTIFICATION  SECTION  (19S) 


178 

184 

190 


GENERALIZATION  SAMPLE  -  CS's  AND  RM's  COMBINED  (N=222) 


EVALUATION  SECTION  (19R) 

JUSTIFICATION  SECTION  (19S) 

EVALUATION  SECTION  (19R)  VS. 
JUSTIFICATION  SECTION  (19S) 


199 

205 

211 


* 


Table  8  (pages  20-21)  contains  a  definition  for  each  of 
the  67  variables.  Variables  1  through  30  are  the  same  as 
Variables  31  through  60  except  for  the  method  of  weighting 
that  was  used.  Therefore,  these  two  sets  of  variables  are 
highly  correlated  within  the  evaluation  section  and  the 
justification  section  of  both  samples.  These  30  correla¬ 
tion  coefficients  are  shown  in  italic  type  in  each  corre¬ 
lation  matrix  for  the  evaluation  section  and  the  justifi¬ 
cation  section. 


CORRELATIONS  AMONG  THE  67  VARIABLES  ON  THE  EVALUATION  SECTION  (19R) 
FOR  THE  CROSS  VALIDATION  SAMPLE  -  AT's  AND  BT's  COMBINED  (N=222) 
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CORRELATIONS  AMONG  THE  67  VARIABLES  ON  THE  EVALUATION  SECTION  (19R) 
FOR  THE  CROSS  VALIDATION  SAMPLE  -  AT’s  AND  BT's  COMBINED  (N=222) 
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CORRELATIONS  AMONG  THE  67  VARIABLES  ON  THE  EVALUATION  SECTION  (19R) 
FOR  THE  CROSS  VALIDATION  SAMPLE  -  AT's  AND  BT's  COMBINED  (N=222) 
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CORRELATIONS  AMONG  THE  67  VARIABLES  ON  THE  EVALUATION  SECTION  (19R) 
FOR  THE  CROSS  VALIDATION  SAMPLE  -  AT's  AND  BT's  COMBINED  (N=222) 
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CORRELATIONS  AMONG  THE  67  VARIABLES  ON  THE  EVALUATION  SECTION  (19R) 
FOR  THE  CROSS  VALIDATION  SAMPLE  -  AT's  AND  BT’s  COMBINED  (N=222) 


182 


ir>  O  r-  CO  O* 
<N  <N  fN  CS  <N  <N 

W  M  w  Ui  W  Ui 


a 


vy  Ln  VO 

2  w  S3 


r*  00  ON 

Qr'i  r> 

W  UJ 


2 


in  <n 


2  2 


m  vo 

3  3 


2 


2 


3 


3 
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CORRELATIONS  AMONG  THE  67  VARIABLES  ON  THE  JUSTIFICATION  SECTION  (19 S) 
FOR  THE  CROSS  VALIDATION  SAMPLE  -  AT's  AND  BT's  COMBINED  (N=222) 
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CORRELATIONS  AMONG  THE  67  VARIABLES  ON  THE  JUSTIFICATION  SECTION  (19S) 
FOR  THE  CROSS  VALIDATION  SAMPLE  -  AT's  AND  BT’s  COMBINED  (N=222) 
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APPENDIX  D 

MANN-WHITNEY  U  TESTS  WITH  ASSOCIATED  z  VALUES*  AND 
t  TESTS  OF  MEAN  DIFFERENCE  ON  ALL  67  VARIABLES 


CROSS  VALIDATION  SAMPLE 

EVALUATION  SECTION 

LOWER  VS.  UPPER  222 

LOWER  VS.  MIDDLE  225 

MIDDLE  VS.  UPPER  228 

JUSTIFICATION  SECTION 

LOWER  VS.  UPPER  231 

LOWER  VS.  MIDDLE  234 

MIDDLE  VS.  UPPER  237 


GENERALIZATION  SAMPLE 

EVALUATION  SECTION 

LOWER  VS.  UPPER  240 

LOWER  VS.  MIDDLE  243 

MIDDLE  VS.  UPPER  246 

JUSTIFICATION  SECTION 

LOWER  VS.  UPPER  249 

LOWER  VS.  MIDDLE  252 

MIDDLE  VS.  UPPER  255 


* 


The  convention  that  was  followed  in  the  Mann-Whitney  U  tests  was  to  enter 
data  into  the  computer  program  first  for  the  lower  criterion  group  in  the 
comparison.  For  those  z  values  that  achieved  statistical  significance,  the 
direction  of  the  difference  is  that  the  higher  criterion  group  evidenced  a 
greater  frequency  or  greater  weighted  frequency  of  the  variable. 


MANN-WHITNEY  U  TESTS  WITH  ASSOCIATED  2  VALUES  AND  t  TESTS  OF  MEAN  DIFFERENCE 
•  ON  ALL  67  VARIABLES  FOR  THE  LOWER  VS.  UPPER  CRITERION  GROUPS  COMPARISON 
OF  THE  CROSS  VALIDATION  SAMPLE  ON  THE  EVALUATION  SECTION  (19R) 
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